Oct 30 23:59:54.911814 ip-10-0-143-53 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 23:59:54.911825 ip-10-0-143-53 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 23:59:54.911832 ip-10-0-143-53 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 23:59:54.912076 ip-10-0-143-53 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:00:04.980942 ip-10-0-143-53 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:00:04.980965 ip-10-0-143-53 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 84ce9d1097a6487186ffae7e2db8654a -- Oct 31 00:02:25.436312 ip-10-0-143-53 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:02:25.771726 ip-10-0-143-53 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:25.771726 ip-10-0-143-53 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:02:25.771726 ip-10-0-143-53 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:25.771726 ip-10-0-143-53 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:25.777066 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.776891 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.777066 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777062 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.777066 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777066 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.777066 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777071 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.777066 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777074 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777077 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777080 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777082 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777085 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777087 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777090 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777093 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777096 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777098 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777100 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777103 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777106 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777108 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777111 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777113 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777116 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777118 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777121 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777123 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.777269 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777126 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777129 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777131 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777134 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777136 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777139 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777141 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777145 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777148 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777150 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777153 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777155 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777158 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777161 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777164 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777167 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777170 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777172 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777175 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.777750 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777178 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777180 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777183 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777186 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777189 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777191 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777194 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777197 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777199 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777202 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777204 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777207 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777210 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777212 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777214 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777217 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777220 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777222 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777225 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777227 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.778207 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777230 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777233 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777235 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777238 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777241 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777243 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777246 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777249 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777252 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777254 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777257 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777261 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777265 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777268 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777273 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777276 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777278 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777281 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777284 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.778713 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777286 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777290 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777293 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777296 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777298 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777301 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777304 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777306 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777309 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777311 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777314 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777317 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777319 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777322 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777324 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777327 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777329 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777332 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777334 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.779176 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777338 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777343 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777347 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777350 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.777354 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778198 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778206 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778209 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778212 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778216 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778218 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778221 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778224 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778226 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778229 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778232 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778235 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778237 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778240 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.779619 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778242 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778245 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778247 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778250 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778253 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778255 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778258 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778260 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778263 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778265 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778267 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778270 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778272 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778275 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778281 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778285 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778288 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778291 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778293 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.780087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778296 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778298 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778301 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778303 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778306 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778308 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778311 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778313 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778316 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778318 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778321 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778324 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778328 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778331 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778333 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778336 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778338 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778341 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778343 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778345 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.780545 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778348 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778350 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778353 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778355 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778358 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778360 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778362 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778367 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778371 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778374 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778376 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778379 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778382 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778385 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778387 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778390 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778393 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778396 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778399 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.781082 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778401 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778404 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778406 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778409 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778413 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778418 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778421 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778423 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778425 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778428 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778430 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778433 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778435 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778438 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778441 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778443 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778446 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778448 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778450 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778453 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.781532 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778457 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778460 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778462 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778465 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778467 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778470 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778472 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778476 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778478 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778481 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778483 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778486 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778488 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.778491 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779010 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779017 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779022 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779025 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779028 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.782041 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779031 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779033 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779036 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779038 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779041 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779044 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779046 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779049 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779052 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779054 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779057 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779059 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779062 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779065 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779067 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779070 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779072 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779075 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779077 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779080 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.782478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779083 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779085 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779087 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779090 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779092 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779095 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779097 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779099 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779102 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779104 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779108 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779110 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779113 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779115 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779118 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779121 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779123 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779125 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.782931 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779129 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779133 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779136 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779140 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779143 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779145 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779148 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779150 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779153 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779155 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779158 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779160 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779163 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779165 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779168 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779170 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779173 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779175 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779178 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779180 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.783378 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779183 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779186 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779188 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779191 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779194 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779197 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779199 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779202 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779204 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779207 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779209 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779212 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779214 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779217 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779219 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779221 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779224 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779226 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779229 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779231 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.783845 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779235 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779237 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779242 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779245 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779247 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779249 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779252 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779254 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779257 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779259 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779261 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779264 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779266 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779269 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779271 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779274 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779276 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779279 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.784436 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779281 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779284 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779286 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.779289 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779381 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779396 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779402 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779407 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779411 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779414 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779419 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779423 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779426 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779429 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779432 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779435 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779438 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779441 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779444 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779447 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779449 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779452 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779457 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:02:25.784878 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779460 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779462 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779465 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779469 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779474 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779478 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779481 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779484 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779487 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779490 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779493 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779496 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779499 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779503 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779506 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779509 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779512 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779515 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779518 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779522 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779525 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779530 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779533 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779537 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:02:25.785434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779541 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779544 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779547 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779549 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779552 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779555 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779558 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779561 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779563 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779566 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779569 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779573 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779577 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779581 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779584 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779587 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779590 2576 flags.go:64] FLAG: --help="false" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779592 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779596 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779599 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779602 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779605 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779608 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:02:25.786094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779611 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779614 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779617 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779619 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779622 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779626 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779630 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779633 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779636 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779639 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779642 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779644 2576 flags.go:64] FLAG: --lock-file="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779647 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779650 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779653 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779658 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779661 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779664 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779666 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779669 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779672 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779676 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779679 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779684 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779687 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:02:25.786643 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779691 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779694 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779697 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779701 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779704 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779707 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779710 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779712 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779720 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779723 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779725 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779728 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779731 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779735 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779738 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779741 2576 flags.go:64] FLAG: --port="10250" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779743 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779747 2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-08b8569304e5bba5d" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779750 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779753 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779756 2576 flags.go:64] FLAG: --register-node="true" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779759 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779763 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779765 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779768 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779771 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:02:25.787245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779775 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779778 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779782 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779784 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779787 2576 flags.go:64] FLAG: --runonce="false" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779790 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779793 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779796 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779799 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779802 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779805 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779808 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779811 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779814 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779817 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779820 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779822 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779826 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779828 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779833 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779838 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779840 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779843 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779848 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779851 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:02:25.787868 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779854 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779857 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779860 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779862 2576 flags.go:64] FLAG: --v="2" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779867 2576 flags.go:64] FLAG: --version="false" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779871 2576 flags.go:64] FLAG: --vmodule="" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779875 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.779878 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780024 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780029 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780032 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780035 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780037 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780040 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780042 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780045 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780047 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780050 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780056 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780059 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780062 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.788564 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780064 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780067 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780070 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780072 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780075 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780078 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780081 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780083 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780086 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780088 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780091 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780095 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780099 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780102 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780105 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780108 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780111 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780114 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.789085 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780118 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780121 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780125 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780127 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780130 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780132 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780135 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780138 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780140 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780143 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780145 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780148 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780151 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780154 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780157 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780159 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780162 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780164 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780167 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780170 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.789515 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780173 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780176 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780178 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780180 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780183 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780185 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780188 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780190 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780193 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780195 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780198 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780200 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780203 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780205 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780209 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780211 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780214 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780216 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780219 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.790037 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780221 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780224 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780226 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780229 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780231 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780233 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780236 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780239 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780241 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780244 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780247 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780249 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780253 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780255 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780258 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780260 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780263 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780265 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780268 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780270 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.790495 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780273 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780275 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780278 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780280 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780282 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780285 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780287 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780291 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780293 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780296 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780298 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780301 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780303 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780306 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780308 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780311 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.790971 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.780337 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780373 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780378 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780381 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780384 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780387 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780389 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780392 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780396 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780398 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780401 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780403 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780406 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780408 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780411 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780413 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780416 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780418 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780420 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780423 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.791367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780426 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780428 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780431 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780433 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780437 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780440 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780442 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780445 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780448 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780450 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780452 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780455 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780458 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780460 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780462 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780464 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780468 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780471 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780473 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780475 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.791843 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780479 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780482 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780484 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780487 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780490 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780492 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780495 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780497 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780499 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780502 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780504 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780507 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780509 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780512 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780514 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780517 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780522 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780526 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780529 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.792409 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780532 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780535 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780538 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780542 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780546 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780549 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780552 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780555 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780558 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780560 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780563 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780566 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780568 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780572 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780575 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780578 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780580 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780583 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780585 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780588 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.792863 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780590 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780593 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780595 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780598 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780600 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780603 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780605 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780608 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780610 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780614 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780617 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780619 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780621 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780624 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780626 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780629 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780631 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780634 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780636 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.793367 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780639 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780641 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780643 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780646 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780649 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780653 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780656 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780659 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.780661 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.787957 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.787991 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.788008 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.788013 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788087 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788094 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788098 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788101 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788103 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788106 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.793805 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788109 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788111 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788114 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788116 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788119 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788122 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788124 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788128 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788131 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788134 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788136 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788139 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788141 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788145 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788147 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788149 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788152 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788160 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788163 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788166 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.794366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788168 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788171 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788173 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788176 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788179 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788182 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788184 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788187 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788190 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788192 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788195 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788197 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788200 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788202 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788204 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788207 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788210 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788212 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788214 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788217 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.794826 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788220 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788222 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788225 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788227 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788230 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788232 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788235 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788237 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788242 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788250 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788253 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788255 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788258 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788260 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788263 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788266 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788268 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788271 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.795314 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788273 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788275 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788278 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788280 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788284 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788287 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788291 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788294 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788297 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788299 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788302 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788304 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788307 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788310 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788313 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788315 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788318 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788320 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.795879 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788323 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788327 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788330 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788333 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788336 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788339 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788347 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788350 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788354 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788358 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788361 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788364 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788366 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788368 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788371 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788373 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788376 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788378 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788381 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.796599 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788383 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788393 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788396 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788400 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.788433 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788625 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788631 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788633 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788636 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788639 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788641 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788644 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788646 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788649 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788651 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788654 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.797366 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788656 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788659 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788661 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788664 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788672 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788675 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788677 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788680 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788682 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788685 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788687 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788690 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788692 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788695 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788697 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788700 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788702 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788705 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788707 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.797744 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788709 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788712 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788714 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788717 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788719 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788722 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788725 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788727 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788730 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788732 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788734 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788737 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788739 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788741 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788744 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788746 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788749 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788751 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788759 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.798321 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788762 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788764 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788767 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788770 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788772 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788775 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788777 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788779 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788782 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788784 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788787 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788789 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788791 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788794 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788796 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788799 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788801 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788803 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788806 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788810 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.798771 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788813 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788816 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788819 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788822 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788825 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788827 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788830 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788832 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788834 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788837 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788839 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788842 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788850 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788852 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788854 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788857 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788859 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788862 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788864 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788866 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.799427 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788869 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788871 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788873 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788876 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788878 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788881 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788883 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788885 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788888 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788890 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788893 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788895 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788898 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788900 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788904 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788907 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.799890 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.788910 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.788933 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789000 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789005 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789007 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789010 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789013 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789015 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789018 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789026 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789029 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789032 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789034 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789037 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789039 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789042 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:25.800317 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789044 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789046 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789049 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789051 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789054 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789056 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789058 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789061 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789063 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789066 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789068 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789071 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789073 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789076 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789079 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789081 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789084 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789086 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789090 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789094 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:25.800693 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789098 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789101 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789103 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789105 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789108 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789110 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789118 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789121 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789123 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789126 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789128 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789130 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789133 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789135 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789138 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789140 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789143 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789145 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789147 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:25.801195 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789150 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789152 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789154 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789157 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789162 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789165 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789167 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789169 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789172 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789174 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789177 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789179 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789182 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789184 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789187 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789189 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789192 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789194 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789197 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:25.801653 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789200 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789208 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789211 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789213 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789215 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789218 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789220 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789223 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789225 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789228 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789230 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789233 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789235 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789237 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789240 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789242 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789245 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789249 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789251 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789254 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:25.802155 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789256 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789258 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789261 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789263 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789265 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789268 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789270 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789273 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789275 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789278 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789280 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789288 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789292 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:25.789295 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.789361 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.791297 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.792049 2576 server.go:1014] "Starting client certificate rotation" Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.792146 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:02:25.802763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.792182 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:02:25.803280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.794198 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:02:25.803280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.798139 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:02:25.809894 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.809875 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:02:25.812492 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.812476 2576 log.go:25] "Validated CRI v1 image API" Oct 31 00:02:25.813376 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.813352 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:02:25.816794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.816770 2576 fs.go:135] Filesystem UUIDs: map[3389db9f-25d5-4298-b0e5-c026e0ba6bbc:/dev/nvme0n1p4 680fe705-c04c-4153-b0db-526a208b930c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 00:02:25.816840 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.816795 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:02:25.821849 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.821739 2576 manager.go:217] Machine: {Timestamp:2026-10-31 00:02:25.820529453 +0000 UTC m=+0.287243572 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100198 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a96320906f023dbf766023fc04ff6 SystemUUID:ec2a9632-0906-f023-dbf7-66023fc04ff6 BootID:84ce9d10-97a6-4871-86ff-ae7e2db8654a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:97:9a:19:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:97:9a:19:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:00:1d:67:a4:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:02:25.821915 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.821852 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:02:25.821952 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.821937 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:02:25.824500 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824475 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:02:25.824636 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824502 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-53.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:02:25.824676 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824639 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:02:25.824676 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824647 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:02:25.824732 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824684 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:02:25.824732 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824700 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:02:25.824794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.824785 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:02:25.828136 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828124 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:02:25.828256 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828247 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:02:25.828818 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828809 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:02:25.828847 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828823 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:02:25.828847 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828834 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:02:25.828847 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828843 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:02:25.828925 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.828855 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:02:25.829321 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.829293 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:02:25.831727 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.831706 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:02:25.834865 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.834850 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:02:25.835686 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835666 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:02:25.835770 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835700 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:02:25.835863 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835852 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:02:25.835891 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835869 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:02:25.835891 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835876 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:02:25.835891 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835882 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:02:25.835891 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835887 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:02:25.835891 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.835892 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:02:25.836308 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.836299 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:02:25.836341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.836309 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:02:25.836341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.836317 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:02:25.836341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.836323 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:02:25.836341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.836335 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:02:25.836555 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.836345 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:02:25.837408 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.837398 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:02:25.837408 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.837408 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:02:25.841190 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.841176 2576 server.go:1263] "Started kubelet" Oct 31 00:02:25.841312 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.841246 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:02:25.841389 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.841269 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:02:25.841438 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.841415 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:02:25.844226 ip-10-0-143-53 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:02:25.844934 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.844748 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:02:25.847013 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.846941 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:02:25.847933 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.847918 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:02:25.851011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.850998 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:02:25.851071 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851019 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:02:25.851142 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.851117 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" Oct 31 00:02:25.851534 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851513 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:02:25.851759 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851740 2576 factory.go:55] Registering systemd factory Oct 31 00:02:25.851841 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851762 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:02:25.851841 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851781 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:02:25.851841 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851794 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:02:25.852001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851897 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:02:25.852001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.851906 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:02:25.852189 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.852169 2576 factory.go:153] Registering CRI-O factory Oct 31 00:02:25.852189 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.852189 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 00:02:25.852321 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.852245 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:02:25.852321 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.852262 2576 factory.go:103] Registering Raw factory Oct 31 00:02:25.852321 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.852277 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 00:02:25.853039 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.853023 2576 manager.go:319] Starting recovery of all containers Oct 31 00:02:25.862150 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.862134 2576 manager.go:324] Recovery completed Oct 31 00:02:25.866736 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.866723 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:25.868200 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868174 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:25.868268 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868214 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:25.868268 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868230 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:25.868721 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868705 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:02:25.868772 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868721 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:02:25.868772 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868738 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:02:25.868772 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.868748 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:25.869416 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.869397 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:25.869502 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.869428 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:25.869502 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.869440 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:25.870831 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.870818 2576 policy_none.go:50] "Start" Oct 31 00:02:25.870896 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.870836 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:02:25.870896 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.870851 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:02:25.872869 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.872854 2576 policy_none.go:44] "Start" Oct 31 00:02:25.912951 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.912765 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:02:25.917142 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917127 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 00:02:25.917194 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.917165 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:02:25.917194 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917176 2576 server.go:87] "Starting device plugin registration server" Oct 31 00:02:25.917454 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917443 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:02:25.917491 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917465 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:02:25.917603 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917582 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:02:25.917696 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917683 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:02:25.917743 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.917698 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:02:25.918215 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.918197 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:02:25.918294 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.918249 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" Oct 31 00:02:25.992386 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.992353 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:02:25.993915 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.993892 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:02:25.993915 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.993920 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:02:25.994071 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.993944 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:02:25.994071 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.994000 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:02:25.997167 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:25.997145 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-53.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:02:25.998184 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:25.997311 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-53.us-west-1.compute.internal.18e3744bdf4a4672 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-53.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-53.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:25.841145458 +0000 UTC m=+0.307859576,LastTimestamp:2026-10-31 00:02:25.841145458 +0000 UTC m=+0.307859576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-53.us-west-1.compute.internal,}" Oct 31 00:02:26.017662 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.017629 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:26.018629 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.018608 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:26.018734 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.018646 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:26.018734 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.018657 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:26.018734 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.018682 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.046946 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:26.046869 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-53.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:02:26.069667 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.069639 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:02:26.069790 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.069728 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tcv2" Oct 31 00:02:26.094985 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.094942 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal"] Oct 31 00:02:26.095106 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.095037 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:26.096028 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.096012 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:26.096123 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.096046 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:26.096123 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.096064 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:26.098374 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.098360 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:26.098521 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.098506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.098560 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.098537 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:26.099154 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.099127 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:26.099252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.099166 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:26.099252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.099177 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:26.099252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.099130 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:26.099252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.099233 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:26.099252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.099249 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:26.101707 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.101693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.101761 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.101717 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:26.102507 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.102488 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:26.102590 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.102524 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:26.102590 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.102541 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:26.126522 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:26.126489 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.128471 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:26.128452 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.153956 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.153922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8844070afaf498134d5a363201a59ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"8844070afaf498134d5a363201a59ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.153956 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.153950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.154174 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.154049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.157496 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.157479 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:26.205732 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.205708 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:26.254497 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254462 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:26.254657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.254657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.254657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8844070afaf498134d5a363201a59ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"8844070afaf498134d5a363201a59ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.254657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254618 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.254657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.254809 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254679 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8844070afaf498134d5a363201a59ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"8844070afaf498134d5a363201a59ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.254910 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.254896 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:26.255822 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.255802 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.255919 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:26.255833 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-53.us-west-1.compute.internal\": node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" Oct 31 00:02:26.351320 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.351232 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.411961 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.411929 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:26.412144 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.412093 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.429438 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.429414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.430509 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.430491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:02:26.469990 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.469953 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:26.792625 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.792550 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:02:26.793194 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.792697 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:02:26.793194 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.792739 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:02:26.793194 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.792739 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:02:26.793194 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:26.792777 2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aec41e53b5c24408ea18b6975af3dda9-0bfa0ebd3c963d0d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-53.us-west-1.compute.internal.18e3744be0e79e76\": read tcp 10.0.143.53:36010->3.133.20.147:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-53.us-west-1.compute.internal.18e3744be0e79e76 default 5408 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-53.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-53.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-53.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:25 +0000 UTC,LastTimestamp:2026-10-31 00:02:26.018661699 +0000 UTC m=+0.485375817,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-53.us-west-1.compute.internal,}" Oct 31 00:02:26.829632 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.829601 2576 apiserver.go:52] "Watching apiserver" Oct 31 00:02:26.851917 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:26.851885 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:02:27.074339 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.074244 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:57:27 +0000 UTC" deadline="2028-07-03 14:23:21.449252061 +0000 UTC" Oct 31 00:02:27.074339 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.074283 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14678h20m54.37497395s" Oct 31 00:02:27.103895 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.103867 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:02:27.104229 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.104212 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5v8zm","openshift-network-diagnostics/network-check-target-2kmrp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal","openshift-multus/multus-vlv5b","openshift-ovn-kubernetes/ovnkube-node-8l9cg","kube-system/global-pull-secret-syncer-68spz","openshift-network-operator/iptables-alerter-6t9z7","kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-kt7ff","openshift-image-registry/node-ca-rlnr6","openshift-multus/multus-additional-cni-plugins-vq95h","openshift-multus/network-metrics-daemon-4slbb","kube-system/konnectivity-agent-4hztx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4"] Oct 31 00:02:27.106568 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.106550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.109041 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.109022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:27.109134 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.109085 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:27.111169 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.111151 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.113438 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.113419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.115479 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.115463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.115537 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.115523 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:27.117526 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.117514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.119679 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.119665 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.121725 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.121711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.124633 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.124615 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.126733 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.126719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:27.126794 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.126778 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:27.129080 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.129063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.131359 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.131339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.154621 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.153809 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:02:27.158395 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.158375 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.158520 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.158489 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.158739 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.158726 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27h5h\"" Oct 31 00:02:27.160205 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-cnibin\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.160283 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.160283 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.160283 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-device-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.160283 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-cnibin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.160413 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-ovn\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160413 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160493 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-multus-daemon-config\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.160493 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-var-lib-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160565 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-etc-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160565 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-kubernetes\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.160632 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-dbus\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.160632 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-slash\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160632 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-env-overrides\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160771 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.160771 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-hostroot\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.160771 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-conf-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.160771 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-netns\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.160771 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-serviceca\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.160947 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.160947 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cl7\" (UniqueName: \"kubernetes.io/projected/7c04250d-a6cd-48e4-97b7-2959641907a7-kube-api-access-99cl7\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.160947 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.160947 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.160947 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-cni-binary-copy\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.160947 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-socket-dir-parent\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.160961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrl9\" (UniqueName: \"kubernetes.io/projected/9b5cca15-e32c-4fb2-8487-ace277115471-kube-api-access-wwrl9\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-conf\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-run\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-tmp\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-registration-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:27.161155 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/386772ab-964a-485d-9489-75e74e1fe1de-agent-certs\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-host\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvtj\" (UniqueName: \"kubernetes.io/projected/9e406f6b-223a-4073-914b-74b5226b09c4-kube-api-access-mpvtj\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p4g\" (UniqueName: \"kubernetes.io/projected/725956a2-d668-4d42-989a-f5aeca0b2288-kube-api-access-74p4g\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsx7\" (UniqueName: \"kubernetes.io/projected/712ccf72-9a61-465b-a2df-495354882144-kube-api-access-mgsx7\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-modprobe-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzn2\" (UniqueName: \"kubernetes.io/projected/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-kube-api-access-5gzn2\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.161394 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-sys-fs\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-system-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-iptables-alerter-script\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/386772ab-964a-485d-9489-75e74e1fe1de-konnectivity-ca\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712ccf72-9a61-465b-a2df-495354882144-hosts-file\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-kubelet\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-lib-modules\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-host-slash\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.161689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-bin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-systemd-units\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-systemd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-bin\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-script-lib\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x978l\" (UniqueName: \"kubernetes.io/projected/1cbf2b1a-032c-4047-81db-998713cc8927-kube-api-access-x978l\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovn-node-metrics-cert\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.161969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-socket-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.162036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-sys\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-host\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-k8s-cni-cncf-io\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-netns\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-multus\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-multus-certs\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-etc-kubernetes\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-node-log\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-var-lib-kubelet\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.162280 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-etc-tuned\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-config\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-systemd\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-kubelet\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-kubelet-config\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-netd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb59b\" (UniqueName: \"kubernetes.io/projected/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-kube-api-access-jb59b\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftp5\" (UniqueName: \"kubernetes.io/projected/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-kube-api-access-pftp5\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712ccf72-9a61-465b-a2df-495354882144-tmp-dir\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-log-socket\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.162586 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysconfig\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.163039 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162626 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:02:27.163039 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162831 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7xfz6\"" Oct 31 00:02:27.163039 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.162931 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.163039 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.163018 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:02:27.163150 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.163051 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.165522 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165486 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:02:27.165522 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165507 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:02:27.165705 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165690 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ddpr\"" Oct 31 00:02:27.165760 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165708 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:02:27.165760 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165723 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.165760 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165752 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.165857 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.165776 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:02:27.169109 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.169092 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:02:27.169200 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.169187 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.169343 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.169329 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.169409 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.169384 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsk29\"" Oct 31 00:02:27.207701 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.207665 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:02:27.207824 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.207745 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vdfrv" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259865 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259888 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259904 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259909 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gvgd9\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259941 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259947 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259955 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.260001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.259960 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:02:27.260364 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260021 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.260364 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260052 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:02:27.260364 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260064 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:27.260364 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260070 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:02:27.260364 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260067 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:02:27.260364 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260083 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgjmv\"" Oct 31 00:02:27.260553 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260402 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt67z\"" Oct 31 00:02:27.260553 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260410 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wr2gb\"" Oct 31 00:02:27.260553 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.260433 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqqqs\"" Oct 31 00:02:27.263744 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-sys\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.263818 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-host\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.263818 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-k8s-cni-cncf-io\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.263818 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-netns\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.263818 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-multus\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-multus-certs\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-host\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-etc-kubernetes\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-sys\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-node-log\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-etc-kubernetes\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263946 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-netns\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-multus-certs\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.263957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-k8s-cni-cncf-io\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264011 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-multus\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-var-lib-kubelet\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264034 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-node-log\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-etc-tuned\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264101 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-var-lib-kubelet\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-config\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-systemd\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-kubelet\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-kubelet-config\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-netd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-systemd\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb59b\" (UniqueName: \"kubernetes.io/projected/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-kube-api-access-jb59b\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264304 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-kubelet\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264312 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-kubelet-config\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftp5\" (UniqueName: \"kubernetes.io/projected/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-kube-api-access-pftp5\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-netd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.264498 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712ccf72-9a61-465b-a2df-495354882144-tmp-dir\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-log-socket\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysconfig\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-cnibin\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264528 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264541 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-log-socket\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264604 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-cnibin\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-device-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-cnibin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264658 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysconfig\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712ccf72-9a61-465b-a2df-495354882144-tmp-dir\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-ovn\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-multus-daemon-config\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-var-lib-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-cnibin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.265212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-etc-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-kubernetes\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-device-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-ovn\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-dbus\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-slash\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-var-lib-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-env-overrides\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.264967 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-config\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-hostroot\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265058 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-etc-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265117 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-hostroot\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-dbus\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265224 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-kubernetes\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265270 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-slash\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.266165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-conf-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-netns\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-serviceca\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cl7\" (UniqueName: \"kubernetes.io/projected/7c04250d-a6cd-48e4-97b7-2959641907a7-kube-api-access-99cl7\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-netns\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-conf-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-cni-binary-copy\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-socket-dir-parent\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrl9\" (UniqueName: \"kubernetes.io/projected/9b5cca15-e32c-4fb2-8487-ace277115471-kube-api-access-wwrl9\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265529 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-socket-dir-parent\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-conf\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-run\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-tmp\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265650 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-registration-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.265714 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/386772ab-964a-485d-9489-75e74e1fe1de-agent-certs\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-conf\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.265812 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:27.765775794 +0000 UTC m=+2.232489920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-host\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.265954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvtj\" (UniqueName: \"kubernetes.io/projected/9e406f6b-223a-4073-914b-74b5226b09c4-kube-api-access-mpvtj\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p4g\" (UniqueName: \"kubernetes.io/projected/725956a2-d668-4d42-989a-f5aeca0b2288-kube-api-access-74p4g\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-run\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsx7\" (UniqueName: \"kubernetes.io/projected/712ccf72-9a61-465b-a2df-495354882144-kube-api-access-mgsx7\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-modprobe-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266087 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-registration-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.267745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzn2\" (UniqueName: \"kubernetes.io/projected/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-kube-api-access-5gzn2\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-sys-fs\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-system-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-iptables-alerter-script\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266275 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-serviceca\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/386772ab-964a-485d-9489-75e74e1fe1de-konnectivity-ca\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266294 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712ccf72-9a61-465b-a2df-495354882144-hosts-file\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-sys-fs\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266421 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-host\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.266480 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.266524 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:27.766508747 +0000 UTC m=+2.233222868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266548 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-system-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266885 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-modprobe-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712ccf72-9a61-465b-a2df-495354882144-hosts-file\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.268460 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-kubelet\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.266995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-lib-modules\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-host-slash\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-bin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-systemd-units\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-systemd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-bin\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267212 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-script-lib\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x978l\" (UniqueName: \"kubernetes.io/projected/1cbf2b1a-032c-4047-81db-998713cc8927-kube-api-access-x978l\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267281 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-systemd-units\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovn-node-metrics-cert\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-socket-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.269239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267402 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-bin\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-bin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-systemd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.267546 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-host-slash\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.268476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-socket-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.268569 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-kubelet\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.268689 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.268943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-env-overrides\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-script-lib\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-cni-binary-copy\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-iptables-alerter-script\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-tmp\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-etc-tuned\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-multus-daemon-config\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269610 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/386772ab-964a-485d-9489-75e74e1fe1de-konnectivity-ca\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.269989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.269653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-lib-modules\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.270685 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.270166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/386772ab-964a-485d-9489-75e74e1fe1de-agent-certs\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.270994 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.270947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovn-node-metrics-cert\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.323486 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.323446 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:27.323486 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.323481 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:27.323486 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.323496 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:27.323927 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:27.323565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:27.823544941 +0000 UTC m=+2.290259070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:27.325461 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.325401 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftp5\" (UniqueName: \"kubernetes.io/projected/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-kube-api-access-pftp5\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.325627 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.325604 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb59b\" (UniqueName: \"kubernetes.io/projected/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-kube-api-access-jb59b\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.325671 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.325614 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cl7\" (UniqueName: \"kubernetes.io/projected/7c04250d-a6cd-48e4-97b7-2959641907a7-kube-api-access-99cl7\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.326844 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.326761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrl9\" (UniqueName: \"kubernetes.io/projected/9b5cca15-e32c-4fb2-8487-ace277115471-kube-api-access-wwrl9\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.327840 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.327813 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzn2\" (UniqueName: \"kubernetes.io/projected/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-kube-api-access-5gzn2\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.328411 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.328386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsx7\" (UniqueName: \"kubernetes.io/projected/712ccf72-9a61-465b-a2df-495354882144-kube-api-access-mgsx7\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.328457 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.328420 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvtj\" (UniqueName: \"kubernetes.io/projected/9e406f6b-223a-4073-914b-74b5226b09c4-kube-api-access-mpvtj\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.329438 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.329413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x978l\" (UniqueName: \"kubernetes.io/projected/1cbf2b1a-032c-4047-81db-998713cc8927-kube-api-access-x978l\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.329528 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.329452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p4g\" (UniqueName: \"kubernetes.io/projected/725956a2-d668-4d42-989a-f5aeca0b2288-kube-api-access-74p4g\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:27.437369 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.437326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:02:27.450228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.450199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlv5b" Oct 31 00:02:27.479160 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.479130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:02:27.481312 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.481291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:27.494933 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.494913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:02:27.506821 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.506788 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8844070afaf498134d5a363201a59ac4.slice/crio-89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5 WatchSource:0}: Error finding container 89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5: Status 404 returned error can't find the container with id 89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5 Oct 31 00:02:27.507537 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.507514 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106eb17b_35db_4dc6_9eb7_9c3c1b58a0d2.slice/crio-091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb WatchSource:0}: Error finding container 091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb: Status 404 returned error can't find the container with id 091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb Oct 31 00:02:27.508328 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.508296 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87458182c04246bddfda72c933944155.slice/crio-41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7 WatchSource:0}: Error finding container 41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7: Status 404 returned error can't find the container with id 41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7 Oct 31 00:02:27.508765 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.508744 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbf2b1a_032c_4047_81db_998713cc8927.slice/crio-8270726706ffc4106c1346d860a132a292eec8e54f506c3288126dd5d1f20695 WatchSource:0}: Error finding container 8270726706ffc4106c1346d860a132a292eec8e54f506c3288126dd5d1f20695: Status 404 returned error can't find the container with id 8270726706ffc4106c1346d860a132a292eec8e54f506c3288126dd5d1f20695 Oct 31 00:02:27.511934 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.511898 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21baed8_0ec6_4331_b9a1_cc6f14dfa3fc.slice/crio-c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16 WatchSource:0}: Error finding container c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16: Status 404 returned error can't find the container with id c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16 Oct 31 00:02:27.513108 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.513003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:02:27.513570 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.513481 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:02:27.521533 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.521514 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499771b6_6b5b_4ad2_80e9_85f61d4d48e4.slice/crio-a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981 WatchSource:0}: Error finding container a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981: Status 404 returned error can't find the container with id a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981 Oct 31 00:02:27.539651 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.539627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:02:27.540745 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.540724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:27.548839 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.548822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:02:27.550641 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.550620 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e406f6b_223a_4073_914b_74b5226b09c4.slice/crio-ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29 WatchSource:0}: Error finding container ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29: Status 404 returned error can't find the container with id ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29 Oct 31 00:02:27.551335 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.551316 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386772ab_964a_485d_9489_75e74e1fe1de.slice/crio-82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54 WatchSource:0}: Error finding container 82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54: Status 404 returned error can't find the container with id 82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54 Oct 31 00:02:27.558799 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:27.558776 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c04250d_a6cd_48e4_97b7_2959641907a7.slice/crio-5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f WatchSource:0}: Error finding container 5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f: Status 404 returned error can't find the container with id 5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f Oct 31 00:02:27.652372 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:27.652287 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:29.234292 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.234268 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:29.247556 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.247529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:29.247673 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.247577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:29.247712 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.247666 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:29.247712 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.247671 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:29.247780 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.247727 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:30.247714258 +0000 UTC m=+3.237225763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:29.247780 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.247740 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:30.247734656 +0000 UTC m=+3.237246161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:29.348229 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.348190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:29.348814 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.348299 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:29.348814 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.348316 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:29.348814 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.348325 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:29.348814 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:29.348377 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:30.348364621 +0000 UTC m=+3.337876127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:29.453816 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.453734 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:29.476104 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.476050 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"8844070afaf498134d5a363201a59ac4","Type":"ContainerStarted","Data":"89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5"} Oct 31 00:02:29.477163 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.477131 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" event={"ID":"7c04250d-a6cd-48e4-97b7-2959641907a7","Type":"ContainerStarted","Data":"5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f"} Oct 31 00:02:29.478191 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.478153 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rlnr6" event={"ID":"499771b6-6b5b-4ad2-80e9-85f61d4d48e4","Type":"ContainerStarted","Data":"a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981"} Oct 31 00:02:29.479318 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.479298 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16"} Oct 31 00:02:29.480309 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.480288 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6t9z7" event={"ID":"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2","Type":"ContainerStarted","Data":"091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb"} Oct 31 00:02:29.481387 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.481370 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hztx" event={"ID":"386772ab-964a-485d-9489-75e74e1fe1de","Type":"ContainerStarted","Data":"82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54"} Oct 31 00:02:29.482452 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.482434 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerStarted","Data":"ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29"} Oct 31 00:02:29.483412 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.483394 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" event={"ID":"1cbf2b1a-032c-4047-81db-998713cc8927","Type":"ContainerStarted","Data":"8270726706ffc4106c1346d860a132a292eec8e54f506c3288126dd5d1f20695"} Oct 31 00:02:29.484355 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.484336 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"87458182c04246bddfda72c933944155","Type":"ContainerStarted","Data":"41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7"} Oct 31 00:02:29.688603 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.688547 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:28 +0000 UTC" deadline="2028-07-23 19:03:05.557359224 +0000 UTC" Oct 31 00:02:29.688603 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:29.688594 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15163h0m35.868767904s" Oct 31 00:02:29.730695 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:02:29.730653 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712ccf72_9a61_465b_a2df_495354882144.slice/crio-68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46 WatchSource:0}: Error finding container 68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46: Status 404 returned error can't find the container with id 68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46 Oct 31 00:02:30.254457 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.254413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:30.254677 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.254492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:30.254677 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.254616 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:30.254677 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.254677 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:32.254658925 +0000 UTC m=+5.244170439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:30.254791 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.254725 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:30.254791 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.254748 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:32.254740974 +0000 UTC m=+5.244252480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:30.354897 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.354849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:30.355554 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.355080 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:30.355554 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.355104 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:30.355554 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.355117 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:30.355554 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.355197 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:32.35516195 +0000 UTC m=+5.344673463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:30.472161 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.472123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:30.472334 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.472298 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:30.472410 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.472395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:30.472492 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.472473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:30.472553 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.472537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:30.472625 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:30.472607 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:30.498594 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.498550 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlv5b" event={"ID":"9b5cca15-e32c-4fb2-8487-ace277115471","Type":"ContainerStarted","Data":"5b84b8341de2e3c5f36ef8877fb30f161950dac97c7ea446e6119147d997c05b"} Oct 31 00:02:30.512455 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.512284 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5v8zm" event={"ID":"712ccf72-9a61-465b-a2df-495354882144","Type":"ContainerStarted","Data":"68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46"} Oct 31 00:02:30.689341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.689291 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:28 +0000 UTC" deadline="2028-07-20 21:10:38.2508873 +0000 UTC" Oct 31 00:02:30.689341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:30.689335 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15093h8m7.561556797s" Oct 31 00:02:32.272033 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:32.271994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:32.272532 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:32.272089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:32.272532 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.272269 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:32.272532 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.272339 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:36.272320271 +0000 UTC m=+9.261831782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:32.272532 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.272409 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:32.272532 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.272443 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:36.272431358 +0000 UTC m=+9.261942867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:32.372657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:32.372614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:32.372840 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.372819 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:32.372929 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.372844 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:32.372929 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.372857 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:32.372929 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.372922 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:36.372903215 +0000 UTC m=+9.362414783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:32.471952 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:32.471875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:32.472100 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.472034 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:32.472165 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:32.472100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:32.472234 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.472166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:32.472289 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:32.472234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:32.472340 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:32.472306 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:33.531057 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:33.530829 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="396d4dbe3813bef756b908d46a280f6b42d682f1859866bc192c5ab8701de3c3" exitCode=0 Oct 31 00:02:33.531629 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:33.531083 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerDied","Data":"396d4dbe3813bef756b908d46a280f6b42d682f1859866bc192c5ab8701de3c3"} Oct 31 00:02:34.471656 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.471620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:34.471656 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.471652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:34.471908 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:34.471776 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:34.471908 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.471840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:34.471908 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:34.471835 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:34.472070 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:34.471924 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:34.541116 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.541030 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hztx" event={"ID":"386772ab-964a-485d-9489-75e74e1fe1de","Type":"ContainerStarted","Data":"593028ee54403bb39f94607244652385f49fafbdae03a55ea0f7eb0abba69952"} Oct 31 00:02:34.545167 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.545140 2576 generic.go:356] "Generic (PLEG): container finished" podID="87458182c04246bddfda72c933944155" containerID="e9c6631722e779d79dce94fc5da9424386a92e8cd9350d5e8745c50ac1b513fe" exitCode=0 Oct 31 00:02:34.545313 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.545234 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"87458182c04246bddfda72c933944155","Type":"ContainerDied","Data":"e9c6631722e779d79dce94fc5da9424386a92e8cd9350d5e8745c50ac1b513fe"} Oct 31 00:02:34.551220 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.551192 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"8844070afaf498134d5a363201a59ac4","Type":"ContainerStarted","Data":"4b1c918fdcaf50e48d161c69a98ff98d0dc0d155be317c8fd2867933c13c4dbc"} Oct 31 00:02:34.658429 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.658168 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4hztx" podStartSLOduration=2.874159061 podStartE2EDuration="7.658149471s" totalImagesPullingTime="4.78399041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:34.657501949 +0000 UTC m=+7.647013480" watchObservedRunningTime="2026-10-31 00:02:34.658149471 +0000 UTC m=+7.647661000" Oct 31 00:02:34.884866 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.884820 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" podStartSLOduration=7.884807104 podStartE2EDuration="7.884807104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:34.771323019 +0000 UTC m=+7.760834549" watchObservedRunningTime="2026-10-31 00:02:34.884807104 +0000 UTC m=+7.874318632" Oct 31 00:02:34.977003 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.976880 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:34.977423 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:34.977403 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:35.561648 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:35.560919 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"87458182c04246bddfda72c933944155","Type":"ContainerStarted","Data":"d3fdba430a26f86054d4f5accecae929c50992ed163f23287673318420f1ad48"} Oct 31 00:02:35.572910 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:35.572871 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rlnr6" event={"ID":"499771b6-6b5b-4ad2-80e9-85f61d4d48e4","Type":"ContainerStarted","Data":"a6048d7596a6535df520b85419da77e593c99f70c68dee2e27cef2ff772d25a0"} Oct 31 00:02:35.677037 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:35.676982 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" podStartSLOduration=8.676945918 podStartE2EDuration="8.676945918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:35.675855228 +0000 UTC m=+8.665366758" watchObservedRunningTime="2026-10-31 00:02:35.676945918 +0000 UTC m=+8.666457445" Oct 31 00:02:35.791047 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:35.790446 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rlnr6" podStartSLOduration=3.053446451 podStartE2EDuration="8.789153137s" totalImagesPullingTime="5.735706686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:35.789011593 +0000 UTC m=+8.778523122" watchObservedRunningTime="2026-10-31 00:02:35.789153137 +0000 UTC m=+8.778664669" Oct 31 00:02:36.308587 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.308167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:36.308587 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.308239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:36.308587 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.308360 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:36.308587 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.308425 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.308406265 +0000 UTC m=+17.297917777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:36.308587 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.308488 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:36.308587 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.308518 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.308508923 +0000 UTC m=+17.298020432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:36.409767 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.409452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:36.409767 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.409632 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:36.409767 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.409647 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:36.409767 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.409655 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:36.409767 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.409713 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.409695061 +0000 UTC m=+17.399206589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:36.472252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.471675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:36.472252 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.471829 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:36.472252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.471940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:36.472252 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.472016 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:36.472252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.472086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:36.472252 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:36.472164 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:36.581031 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.580956 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" event={"ID":"7c04250d-a6cd-48e4-97b7-2959641907a7","Type":"ContainerStarted","Data":"012a06d42e486ee14b8d415e36c67924b47aac10a64e89a51456db58a756fe38"} Oct 31 00:02:36.581544 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.581522 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:02:36.827813 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:36.827777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:37.087105 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:37.087071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:37.965251 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:37.964536 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:02:38.402437 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.402297 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:02:37.964550971Z","UUID":"a94e25fe-991f-4385-8693-456d9ad96033","Handler":null,"Name":"","Endpoint":""} Oct 31 00:02:38.404110 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.404089 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:02:38.404231 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.404122 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:02:38.471401 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.471371 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:38.471537 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:38.471515 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:38.471590 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.471583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:38.471646 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:38.471628 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:38.471692 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.471673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:38.471758 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:38.471736 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:38.587212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.587012 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5v8zm" event={"ID":"712ccf72-9a61-465b-a2df-495354882144","Type":"ContainerStarted","Data":"d232ec47956a1fc203688c4c6c4c131c586d5430a7052cab4a68f714e4636be9"} Oct 31 00:02:38.589578 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.589545 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerStarted","Data":"9655bc37dfadbdedcebba902e078c1c7e8e2628c288ec34116a28f025b976b67"} Oct 31 00:02:38.592444 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.592357 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" event={"ID":"7c04250d-a6cd-48e4-97b7-2959641907a7","Type":"ContainerStarted","Data":"f8828ef0025b58f77c659c152de788ab5bac003c6cdc28bf5de2b64f3783682a"} Oct 31 00:02:38.593825 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.593789 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlv5b" event={"ID":"9b5cca15-e32c-4fb2-8487-ace277115471","Type":"ContainerStarted","Data":"8ae937f5aa047b84188a242c3961eb01949c1e0e84951530995dc4baf64de47a"} Oct 31 00:02:38.625332 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.625297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:38.703402 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.703354 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5v8zm" podStartSLOduration=3.060593802 podStartE2EDuration="11.703341073s" totalImagesPullingTime="8.642747271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:38.703099373 +0000 UTC m=+11.692610893" watchObservedRunningTime="2026-10-31 00:02:38.703341073 +0000 UTC m=+11.692852600" Oct 31 00:02:38.773095 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.773053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:38.820766 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:38.820708 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vlv5b" podStartSLOduration=3.678144513 podStartE2EDuration="11.820688529s" totalImagesPullingTime="8.142544016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:38.820058905 +0000 UTC m=+11.809570446" watchObservedRunningTime="2026-10-31 00:02:38.820688529 +0000 UTC m=+11.810200062" Oct 31 00:02:39.598339 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:39.598169 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6t9z7" event={"ID":"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2","Type":"ContainerStarted","Data":"fd530b52aec39aa470e31ebddd1e1d727c8afc169a8068671645d2968c3339ca"} Oct 31 00:02:39.601218 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:39.601100 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="9655bc37dfadbdedcebba902e078c1c7e8e2628c288ec34116a28f025b976b67" exitCode=0 Oct 31 00:02:39.601448 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:39.601419 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerDied","Data":"9655bc37dfadbdedcebba902e078c1c7e8e2628c288ec34116a28f025b976b67"} Oct 31 00:02:39.713732 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:39.713645 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6t9z7" podStartSLOduration=4.038454134 podStartE2EDuration="12.713630263s" totalImagesPullingTime="8.675176129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:39.713358633 +0000 UTC m=+12.702870173" watchObservedRunningTime="2026-10-31 00:02:39.713630263 +0000 UTC m=+12.703141791" Oct 31 00:02:40.243901 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.243868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:40.388844 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.388807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:40.472216 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.472123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:40.472216 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.472198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:40.472216 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.472197 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:40.472463 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:40.472281 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:40.472463 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:40.472363 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:40.472463 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:40.472438 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:40.604837 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.604797 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" event={"ID":"1cbf2b1a-032c-4047-81db-998713cc8927","Type":"ContainerStarted","Data":"14182838dedb7f14d4550a43dbd3dd399ca56e4d8de4137d59ea05c1b925f072"} Oct 31 00:02:40.607245 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.607219 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" event={"ID":"7c04250d-a6cd-48e4-97b7-2959641907a7","Type":"ContainerStarted","Data":"7f034f7d78216e74aa9bbd687abd2a03fb6624f080f99184d7e112f7a204e648"} Oct 31 00:02:40.721890 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.721826 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" podStartSLOduration=2.51428752 podStartE2EDuration="13.721808303s" totalImagesPullingTime="11.207520783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:40.721261714 +0000 UTC m=+13.710773242" watchObservedRunningTime="2026-10-31 00:02:40.721808303 +0000 UTC m=+13.711319829" Oct 31 00:02:40.839708 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:40.839626 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" podStartSLOduration=3.051298299 podStartE2EDuration="13.839609122s" totalImagesPullingTime="10.788310823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:40.837816725 +0000 UTC m=+13.827328238" watchObservedRunningTime="2026-10-31 00:02:40.839609122 +0000 UTC m=+13.829120650" Oct 31 00:02:41.430070 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:41.430007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:41.599324 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:41.599249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:41.612198 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:41.612146 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="52c5d9d3e2fe18c3fd2520e65f2872a6f0b4414b5deb1c245f5bdc7326116011" exitCode=0 Oct 31 00:02:41.612712 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:41.612230 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerDied","Data":"52c5d9d3e2fe18c3fd2520e65f2872a6f0b4414b5deb1c245f5bdc7326116011"} Oct 31 00:02:42.471924 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:42.471889 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:42.472096 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:42.471940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:42.472096 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:42.471905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:42.472096 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:42.472044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:42.472240 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:42.472100 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:42.472240 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:42.472196 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:43.619043 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:43.618770 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="66d76739c1b41b3fb6c107692cef862f9c3f32440a61c501e91fde2b45d0903d" exitCode=0 Oct 31 00:02:43.619632 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:43.618850 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerDied","Data":"66d76739c1b41b3fb6c107692cef862f9c3f32440a61c501e91fde2b45d0903d"} Oct 31 00:02:43.957130 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:43.957053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:44.168804 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.168771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:44.372920 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.372830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:44.372920 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.372884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:44.373098 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.372993 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:44.373098 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.373056 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:00.373042163 +0000 UTC m=+33.362553669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:44.373098 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.372993 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:44.373221 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.373107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:00.37309584 +0000 UTC m=+33.362607360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:44.435455 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.435417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:44.472035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.472000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:44.472035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.472030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:44.472260 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.472147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:44.472260 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.472148 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:44.472370 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.472291 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:44.472425 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.472389 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:44.473115 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.473102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:44.473232 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.473220 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:44.473265 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.473235 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:44.473265 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.473246 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:44.473342 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:44.473285 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:03:00.473271808 +0000 UTC m=+33.462783315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:44.678276 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.678227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:44.836719 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:44.836687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:45.015275 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.015245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:45.162911 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.162658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:45.630802 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.630612 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"ae252bfbcbad59d2506005b2c1e0bd5d300e316ab5be97585019a6b39c297a74"} Oct 31 00:02:45.630802 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.630738 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"f57601d2f2ca9e383cec09f13186d10e758e0850937eb605139a4950e067596d"} Oct 31 00:02:45.630802 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.630769 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"8d0ee41b3cefddcf3a974ceab85260ea1c224b0dbe654344aaa6126365e94029"} Oct 31 00:02:45.630802 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.630802 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"2ec50c39bce557f76c4276e2e9c08ce05b735754ef0422a73d166f7f216de192"} Oct 31 00:02:45.631027 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.630838 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"3ff967155d8190e2e676d3937af89df3ebb0192fd4aae38b7702d7f7de6831af"} Oct 31 00:02:45.631027 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.630866 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"012280fcba6dca93693252142c73139b70e14e49f86fa61d6f4b127d60bff2c7"} Oct 31 00:02:45.895143 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:45.895055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:46.060109 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.060077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:46.471524 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.471492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:46.471666 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.471576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:46.471666 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.471590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:46.471760 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:46.471728 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:46.471814 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:46.471794 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:46.471898 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:46.471883 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:46.603796 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.603761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:46.635359 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.635327 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="a3621205af7df5e1ad1745648d6159baa263f3eca2adec5b9a8fa0cf25296c07" exitCode=0 Oct 31 00:02:46.635510 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:46.635395 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerDied","Data":"a3621205af7df5e1ad1745648d6159baa263f3eca2adec5b9a8fa0cf25296c07"} Oct 31 00:02:47.640274 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:47.640237 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"4e38dfc73a9ba7ab304d9b618d474290ca85e6509788e20a7cbd348420489a4c"} Oct 31 00:02:47.642893 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:47.642866 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="8ec32c713d3c4392704480559c7c8fb55fafd3040cc822d063a54c255434f151" exitCode=0 Oct 31 00:02:47.642983 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:47.642916 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerDied","Data":"8ec32c713d3c4392704480559c7c8fb55fafd3040cc822d063a54c255434f151"} Oct 31 00:02:48.472052 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.472018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:48.472052 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.472051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:48.472320 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:48.472124 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:48.472320 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.472220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:48.472467 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:48.472317 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:48.472467 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:48.472414 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:48.584400 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.584362 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:48.584549 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.584491 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:02:48.584909 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.584896 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:02:48.648513 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.648476 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vq95h" event={"ID":"9e406f6b-223a-4073-914b-74b5226b09c4","Type":"ContainerStarted","Data":"9c60d5c8cca2527fd1318a2c6d4e5fcab8205cea87fcab06a7159924bd5891f2"} Oct 31 00:02:48.821408 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:48.821317 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vq95h" podStartSLOduration=8.9161441 podStartE2EDuration="21.821302639s" totalImagesPullingTime="12.905158539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:48.820792242 +0000 UTC m=+21.810303770" watchObservedRunningTime="2026-10-31 00:02:48.821302639 +0000 UTC m=+21.810814168" Oct 31 00:02:49.080247 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:49.080152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:49.654331 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:49.654119 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" event={"ID":"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc","Type":"ContainerStarted","Data":"462e51cff68bb2c4300063fc9802206ad31a65915b728fc6efa8eb29727ed4cb"} Oct 31 00:02:49.929622 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:49.929532 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" podStartSLOduration=7.244442029 podStartE2EDuration="22.929514151s" totalImagesPullingTime="15.685072122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:49.879953114 +0000 UTC m=+22.869464642" watchObservedRunningTime="2026-10-31 00:02:49.929514151 +0000 UTC m=+22.919025673" Oct 31 00:02:50.113001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.112969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:50.350450 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.350382 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:50.471707 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.471666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:50.471871 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.471748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:50.471871 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.471836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:50.471871 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:50.471855 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:50.471965 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:50.471900 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:50.472017 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:50.471978 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:50.656716 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.656630 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:50.656716 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.656703 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:50.672843 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.672816 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:50.672989 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:50.672955 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:02:52.471444 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:52.471413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:52.471843 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:52.471417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:52.471843 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:52.471517 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:52.471843 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:52.471417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:52.471843 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:52.471607 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:52.471843 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:52.471661 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:52.658288 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:52.658256 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5v8zm_712ccf72-9a61-465b-a2df-495354882144/dns-node-resolver/0.log" Oct 31 00:02:53.776091 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:53.776052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:54.064978 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.064880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:54.220921 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.220886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:54.264834 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.264805 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rlnr6_499771b6-6b5b-4ad2-80e9-85f61d4d48e4/node-ca/0.log" Oct 31 00:02:54.381577 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.381479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:54.472344 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.472309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:54.472490 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.472369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:54.472490 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.472463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:54.472490 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:54.472473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:54.472579 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:54.472543 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:54.472614 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:54.472603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:54.525489 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:54.525445 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:55.715049 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:55.715017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:55.885068 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:55.885028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:56.035087 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.034994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:56.207681 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.207631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:56.405233 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.405129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:56.471476 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.471437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:56.471476 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.471462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:56.471691 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:56.471549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:56.471691 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:56.471669 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:56.471773 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.471725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:56.471820 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:56.471803 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:56.550598 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:56.550564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:57.632902 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:57.632871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:57.834199 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:57.834148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:57.995595 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:57.995512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:58.169146 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:58.169101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:58.312506 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:58.312408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:58.460156 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:58.460120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:58.472262 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:58.472234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:02:58.472390 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:58.472275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:02:58.472390 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:58.472366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:02:58.472390 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:58.472376 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:02:58.472532 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:58.472472 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:02:58.472532 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:02:58.472503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:02:59.682886 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:59.682854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:59.836338 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:02:59.836304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:00.011896 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.011812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:00.209707 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.209668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:00.364962 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.364885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:00.383889 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.383863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:00.383987 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.383903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:00.384023 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.383988 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:03:00.384023 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.383989 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:03:00.384089 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.384036 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs podName:725956a2-d668-4d42-989a-f5aeca0b2288 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:32.384023545 +0000 UTC m=+65.373535065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs") pod "network-metrics-daemon-4slbb" (UID: "725956a2-d668-4d42-989a-f5aeca0b2288") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:03:00.384089 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.384048 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret podName:1bc7e49e-8890-47c1-a64f-25328161fde3 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:32.384042542 +0000 UTC m=+65.373554050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret") pod "global-pull-secret-syncer-68spz" (UID: "1bc7e49e-8890-47c1-a64f-25328161fde3") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:03:00.472123 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.472096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:00.472123 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.472124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:00.472371 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.472251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:00.472371 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.472254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:00.472371 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.472331 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:00.472466 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.472400 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:00.484333 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.484315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:00.484468 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.484455 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:03:00.484508 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.484475 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:03:00.484508 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.484483 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dkwr7 for pod openshift-network-diagnostics/network-check-target-2kmrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:03:00.484568 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:00.484533 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7 podName:38f2d18c-7a3f-498f-b66e-b104cbab0e6c nodeName:}" failed. No retries permitted until 2026-10-31 00:03:32.484516351 +0000 UTC m=+65.474027871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkwr7" (UniqueName: "kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7") pod "network-check-target-2kmrp" (UID: "38f2d18c-7a3f-498f-b66e-b104cbab0e6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:03:00.507561 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:00.507537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:01.610079 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:01.610041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:01.783071 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:01.783033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:01.925849 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:01.925750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:02.068042 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:02.068008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:02.218811 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:02.218720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:02.409442 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:02.409408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:02.472239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:02.472153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:02.472239 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:02.472186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:02.472405 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:02.472254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:02.472405 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:02.472295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:02.472405 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:02.472382 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:02.472504 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:02.472443 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:03.608880 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:03.608842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:03.758258 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:03.758218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:03.904248 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:03.904149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:04.050122 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:04.050083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:04.207480 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:04.207400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:04.406825 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:04.406788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:04.472212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:04.472127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:04.472212 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:04.472153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:04.472370 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:04.472254 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:04.472370 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:04.472274 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:04.472370 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:04.472354 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:04.472459 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:04.472425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:06.472139 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:06.472104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:06.472139 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:06.472134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:06.472660 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:06.472105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:06.472660 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:06.472222 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:06.472660 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:06.472310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:06.472660 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:06.472403 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:07.869275 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:07.869243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:08.040594 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.040567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:08.471401 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.471368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:08.471401 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.471389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:08.471640 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.471392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:08.471640 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:08.471491 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:08.471640 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:08.471551 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:08.471640 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:08.471625 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:08.472401 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.472377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:08.675313 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.675280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:08.881350 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:08.881318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:09.029477 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:09.029443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:09.685615 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:09.685583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:09.869370 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:09.869285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:10.026783 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:10.026752 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2kmrp"] Oct 31 00:03:10.027347 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:10.027320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:10.027477 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:10.027448 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:10.030332 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:10.030307 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-68spz"] Oct 31 00:03:10.030582 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:10.030565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:10.030799 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:10.030778 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:10.031079 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:10.031052 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4slbb"] Oct 31 00:03:10.031171 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:10.031154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:10.031322 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:10.031281 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:11.404840 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:11.404653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:11.471476 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:11.471444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:11.471476 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:11.471475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:11.471662 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:11.471483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:11.471662 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:11.471562 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:11.471734 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:11.471673 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:11.471768 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:11.471736 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:11.547419 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:11.547388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:13.028703 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.028663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:13.172557 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.172521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:13.319733 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.319646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:13.470417 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.470389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:13.472349 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.472332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:13.472459 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.472364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:13.472459 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.472337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:13.472555 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:13.472450 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kmrp" podUID="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" Oct 31 00:03:13.472555 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:13.472525 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4slbb" podUID="725956a2-d668-4d42-989a-f5aeca0b2288" Oct 31 00:03:13.472657 ip-10-0-143-53 kubenswrapper[2576]: E1031 00:03:13.472585 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-68spz" podUID="1bc7e49e-8890-47c1-a64f-25328161fde3" Oct 31 00:03:13.629374 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:13.629298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:14.739210 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.739170 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeReady" Oct 31 00:03:14.739515 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.739310 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:03:14.893425 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.893393 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p2dzg"] Oct 31 00:03:14.895094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.895079 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q4dds"] Oct 31 00:03:14.895247 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.895227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:14.896800 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.896778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:14.950794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.950764 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nxhq2"] Oct 31 00:03:14.952508 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.952485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:14.953655 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953634 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:03:14.953794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953671 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:03:14.953794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953671 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:03:14.953794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953777 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:03:14.953794 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953790 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:03:14.953950 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953797 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4gwp\"" Oct 31 00:03:14.953950 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.953849 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtt85\"" Oct 31 00:03:14.973063 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.973012 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2dzg"] Oct 31 00:03:14.976533 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.976509 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4dds"] Oct 31 00:03:14.988962 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.988938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dc42da-f207-4a5c-a481-6d5a65363802-config-volume\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:14.989069 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.988965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e16f2f0-c342-414d-a8c8-5f68528c78b8-crio-socket\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:14.989069 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.988993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjm8\" (UniqueName: \"kubernetes.io/projected/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-api-access-ljjm8\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:14.989134 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.989121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqh9g\" (UniqueName: \"kubernetes.io/projected/f6dc42da-f207-4a5c-a481-6d5a65363802-kube-api-access-bqh9g\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:14.989166 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.989147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6dc42da-f207-4a5c-a481-6d5a65363802-metrics-tls\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:14.989217 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.989172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dc42da-f207-4a5c-a481-6d5a65363802-tmp-dir\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:14.989249 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.989213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:14.989289 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.989275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e16f2f0-c342-414d-a8c8-5f68528c78b8-data-volume\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:14.989320 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:14.989305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e16f2f0-c342-414d-a8c8-5f68528c78b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.022743 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.022713 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nxhq2"] Oct 31 00:03:15.053675 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.053642 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:03:15.053804 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.053723 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:03:15.053804 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.053743 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:03:15.053804 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.053782 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:03:15.053895 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.053852 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hplth\"" Oct 31 00:03:15.089941 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.089918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dc42da-f207-4a5c-a481-6d5a65363802-config-volume\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.090018 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.089944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e16f2f0-c342-414d-a8c8-5f68528c78b8-crio-socket\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090018 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.089964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955eeb9d-5b8c-473a-b0cf-69d6672353bf-cert\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.090018 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.089979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6fr\" (UniqueName: \"kubernetes.io/projected/955eeb9d-5b8c-473a-b0cf-69d6672353bf-kube-api-access-db6fr\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.090103 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjm8\" (UniqueName: \"kubernetes.io/projected/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-api-access-ljjm8\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090103 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqh9g\" (UniqueName: \"kubernetes.io/projected/f6dc42da-f207-4a5c-a481-6d5a65363802-kube-api-access-bqh9g\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.090174 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6dc42da-f207-4a5c-a481-6d5a65363802-metrics-tls\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.090174 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dc42da-f207-4a5c-a481-6d5a65363802-tmp-dir\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.090296 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e16f2f0-c342-414d-a8c8-5f68528c78b8-crio-socket\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090296 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090296 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e16f2f0-c342-414d-a8c8-5f68528c78b8-data-volume\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090296 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e16f2f0-c342-414d-a8c8-5f68528c78b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090555 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090533 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dc42da-f207-4a5c-a481-6d5a65363802-tmp-dir\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.090685 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e16f2f0-c342-414d-a8c8-5f68528c78b8-data-volume\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.090760 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dc42da-f207-4a5c-a481-6d5a65363802-config-volume\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.090826 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.090804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.094166 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.094141 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e16f2f0-c342-414d-a8c8-5f68528c78b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.094285 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.094204 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6dc42da-f207-4a5c-a481-6d5a65363802-metrics-tls\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.150956 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.150930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjm8\" (UniqueName: \"kubernetes.io/projected/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-api-access-ljjm8\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.151202 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.151165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqh9g\" (UniqueName: \"kubernetes.io/projected/f6dc42da-f207-4a5c-a481-6d5a65363802-kube-api-access-bqh9g\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.191519 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.191487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955eeb9d-5b8c-473a-b0cf-69d6672353bf-cert\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.191519 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.191520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6fr\" (UniqueName: \"kubernetes.io/projected/955eeb9d-5b8c-473a-b0cf-69d6672353bf-kube-api-access-db6fr\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.193863 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.193846 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955eeb9d-5b8c-473a-b0cf-69d6672353bf-cert\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.207724 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.207704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:15.214537 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.214523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:15.257950 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.257914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6fr\" (UniqueName: \"kubernetes.io/projected/955eeb9d-5b8c-473a-b0cf-69d6672353bf-kube-api-access-db6fr\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.261458 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.261422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:15.472455 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.472419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:15.472455 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.472440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:15.472667 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.472521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:15.524908 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.524880 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:03:15.525260 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.525242 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:03:15.525343 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.525273 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:03:15.525343 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.525292 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\"" Oct 31 00:03:15.525416 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.525360 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\"" Oct 31 00:03:15.525450 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.525427 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\"" Oct 31 00:03:15.525516 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.525506 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:03:15.598070 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.598041 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2dzg"] Oct 31 00:03:15.600987 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.600965 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4dds"] Oct 31 00:03:15.602430 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:15.602404 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dc42da_f207_4a5c_a481_6d5a65363802.slice/crio-69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85 WatchSource:0}: Error finding container 69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85: Status 404 returned error can't find the container with id 69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85 Oct 31 00:03:15.603877 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:15.603856 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e16f2f0_c342_414d_a8c8_5f68528c78b8.slice/crio-e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6 WatchSource:0}: Error finding container e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6: Status 404 returned error can't find the container with id e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6 Oct 31 00:03:15.645706 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.645683 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nxhq2"] Oct 31 00:03:15.649276 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:15.649255 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955eeb9d_5b8c_473a_b0cf_69d6672353bf.slice/crio-11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544 WatchSource:0}: Error finding container 11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544: Status 404 returned error can't find the container with id 11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544 Oct 31 00:03:15.713341 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.713312 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nxhq2" event={"ID":"955eeb9d-5b8c-473a-b0cf-69d6672353bf","Type":"ContainerStarted","Data":"11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544"} Oct 31 00:03:15.717911 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.717883 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4dds" event={"ID":"5e16f2f0-c342-414d-a8c8-5f68528c78b8","Type":"ContainerStarted","Data":"278675bba2c490e02fc76542d6fabe730a3cc766d0b069b8dba70232ba5465bf"} Oct 31 00:03:15.718035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.717952 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4dds" event={"ID":"5e16f2f0-c342-414d-a8c8-5f68528c78b8","Type":"ContainerStarted","Data":"e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6"} Oct 31 00:03:15.719013 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:15.718992 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2dzg" event={"ID":"f6dc42da-f207-4a5c-a481-6d5a65363802","Type":"ContainerStarted","Data":"69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85"} Oct 31 00:03:17.727504 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:17.727467 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4dds" event={"ID":"5e16f2f0-c342-414d-a8c8-5f68528c78b8","Type":"ContainerStarted","Data":"141a29bead88518c47e2d80911a90540fcc617ad15fcc8737f8e24433a20d76c"} Oct 31 00:03:18.357977 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.357890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:18.527348 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.527262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:18.674597 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.674567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:18.732376 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.732344 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4dds" event={"ID":"5e16f2f0-c342-414d-a8c8-5f68528c78b8","Type":"ContainerStarted","Data":"ed4015a78698570b159a1bc53d1698cee1030941e557fdfb2951736638a47bfa"} Oct 31 00:03:18.734684 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.734659 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2dzg" event={"ID":"f6dc42da-f207-4a5c-a481-6d5a65363802","Type":"ContainerStarted","Data":"b3eb3a388045f9048d263c073bd74267deebff3282134964099a2e285a599d32"} Oct 31 00:03:18.734813 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.734710 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2dzg" event={"ID":"f6dc42da-f207-4a5c-a481-6d5a65363802","Type":"ContainerStarted","Data":"1b243d2e7199d7894026d4ac5078ef9ea0ea92ce719b3715c51e9eab98ce1a37"} Oct 31 00:03:18.734813 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.734804 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:18.850442 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.850381 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q4dds" podStartSLOduration=1.953636859 podStartE2EDuration="4.850362654s" totalImagesPullingTime="2.896725795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:03:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:18.849277351 +0000 UTC m=+51.838788880" watchObservedRunningTime="2026-10-31 00:03:18.850362654 +0000 UTC m=+51.839874183" Oct 31 00:03:18.867919 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.867887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:18.966050 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:18.965997 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-p2dzg" podStartSLOduration=2.876930417 podStartE2EDuration="4.965982931s" totalImagesPullingTime="2.089052514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:18.965288173 +0000 UTC m=+51.954799703" watchObservedRunningTime="2026-10-31 00:03:18.965982931 +0000 UTC m=+51.955494487" Oct 31 00:03:19.017303 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.017275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.065908 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.065871 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg"] Oct 31 00:03:19.074409 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.074379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.129094 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.129065 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:03:19.129268 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.129102 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:03:19.129606 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.129589 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:03:19.129710 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.129615 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:03:19.129710 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.129669 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8ntdh\"" Oct 31 00:03:19.129832 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.129815 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:03:19.130274 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.130255 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg"] Oct 31 00:03:19.173877 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.173787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.179331 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.179308 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-67jm8"] Oct 31 00:03:19.181057 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.181038 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr"] Oct 31 00:03:19.181224 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.181206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.183309 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.183288 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.224237 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.224214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.224345 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.224255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4pb\" (UniqueName: \"kubernetes.io/projected/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-kube-api-access-bg4pb\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.224345 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.224333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.224427 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.224413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.233728 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.233706 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:03:19.233816 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.233740 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sm2z2\"" Oct 31 00:03:19.233871 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.233815 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:03:19.233871 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.233849 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:03:19.235230 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.235213 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:03:19.235230 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.235229 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:03:19.235354 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.235248 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wcptn\"" Oct 31 00:03:19.235354 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.235249 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:03:19.280143 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.280116 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr"] Oct 31 00:03:19.317641 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.317606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.324685 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.324769 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-root\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.324769 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.324874 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-metrics-client-ca\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.324931 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.324982 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.325032 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.324992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztr5\" (UniqueName: \"kubernetes.io/projected/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-api-access-jztr5\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.325032 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-tls\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.325097 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-wtmp\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.325134 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.325193 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-textfile\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.325230 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a61bcee6-5eb5-44b0-af5b-81bf788bf136-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.325259 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlbw\" (UniqueName: \"kubernetes.io/projected/a0a53051-3a82-499c-bd29-9ccf93a535bc-kube-api-access-5hlbw\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.325310 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.325409 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.325461 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.325461 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-sys\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.325551 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.325647 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4pb\" (UniqueName: \"kubernetes.io/projected/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-kube-api-access-bg4pb\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.326010 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.325990 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.327636 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.327608 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.327766 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.327748 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.386755 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.386732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4pb\" (UniqueName: \"kubernetes.io/projected/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-kube-api-access-bg4pb\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.426889 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.426821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-textfile\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.426889 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.426865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a61bcee6-5eb5-44b0-af5b-81bf788bf136-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427045 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.426981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlbw\" (UniqueName: \"kubernetes.io/projected/a0a53051-3a82-499c-bd29-9ccf93a535bc-kube-api-access-5hlbw\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427045 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427045 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-sys\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427265 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427265 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-sys\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427265 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427198 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-textfile\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427265 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427427 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-root\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427427 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a61bcee6-5eb5-44b0-af5b-81bf788bf136-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427427 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427427 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-metrics-client-ca\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427607 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427369 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-root\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427607 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427607 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427607 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztr5\" (UniqueName: \"kubernetes.io/projected/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-api-access-jztr5\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427607 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-tls\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427607 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-wtmp\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.427862 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427862 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.427862 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-wtmp\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.428005 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.427957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.428219 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.428172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-metrics-client-ca\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.429821 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.429793 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.429918 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.429799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.430083 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.430068 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.430252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.430232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-tls\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.487602 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.487562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlbw\" (UniqueName: \"kubernetes.io/projected/a0a53051-3a82-499c-bd29-9ccf93a535bc-kube-api-access-5hlbw\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.488008 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.487991 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztr5\" (UniqueName: \"kubernetes.io/projected/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-api-access-jztr5\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.494829 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.494810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:19.499130 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.499116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:19.501567 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.501500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.505542 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:19.505521 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a53051_3a82_499c_bd29_9ccf93a535bc.slice/crio-80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4 WatchSource:0}: Error finding container 80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4: Status 404 returned error can't find the container with id 80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4 Oct 31 00:03:19.685932 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.685864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:19.712149 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.712117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:19.739698 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.739601 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67jm8" event={"ID":"a0a53051-3a82-499c-bd29-9ccf93a535bc","Type":"ContainerStarted","Data":"80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4"} Oct 31 00:03:19.741782 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.741750 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nxhq2" event={"ID":"955eeb9d-5b8c-473a-b0cf-69d6672353bf","Type":"ContainerStarted","Data":"d741b294cd4eb453fa791b18fd1c74618b0458d75f9f4877202c84eb2f4edaef"} Oct 31 00:03:19.857668 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.857619 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nxhq2" podStartSLOduration=2.624874709 podStartE2EDuration="5.857604129s" totalImagesPullingTime="3.23272942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:19.856420784 +0000 UTC m=+52.845932314" watchObservedRunningTime="2026-10-31 00:03:19.857604129 +0000 UTC m=+52.847115782" Oct 31 00:03:19.918157 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:19.918125 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr"] Oct 31 00:03:19.921652 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:19.921615 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61bcee6_5eb5_44b0_af5b_81bf788bf136.slice/crio-9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1 WatchSource:0}: Error finding container 9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1: Status 404 returned error can't find the container with id 9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1 Oct 31 00:03:20.052671 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.052642 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg"] Oct 31 00:03:20.055740 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:20.055712 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce2530e_5ad4_4eec_ad2c_58bca81cae53.slice/crio-2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5 WatchSource:0}: Error finding container 2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5: Status 404 returned error can't find the container with id 2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5 Oct 31 00:03:20.490099 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.489996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:20.695693 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.695659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:20.747620 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.747556 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" event={"ID":"1ce2530e-5ad4-4eec-ad2c-58bca81cae53","Type":"ContainerStarted","Data":"64db97ad4af2d41fd64953da5bb44817812396a2935d5943da4769ad9ac0c9ac"} Oct 31 00:03:20.748015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.747657 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" event={"ID":"1ce2530e-5ad4-4eec-ad2c-58bca81cae53","Type":"ContainerStarted","Data":"601a3930f581c0cf1025da0637a8dd0ed1ec0ab35c7802554187174fe5419278"} Oct 31 00:03:20.748015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.747692 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" event={"ID":"1ce2530e-5ad4-4eec-ad2c-58bca81cae53","Type":"ContainerStarted","Data":"2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5"} Oct 31 00:03:20.749042 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:20.749012 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" event={"ID":"a61bcee6-5eb5-44b0-af5b-81bf788bf136","Type":"ContainerStarted","Data":"9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1"} Oct 31 00:03:21.358219 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.358121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:21.400932 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.400895 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:03:21.403621 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.403598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.456022 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.455983 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:03:21.456022 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.455992 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:03:21.456250 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.455992 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:03:21.456368 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.456351 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-27lg5\"" Oct 31 00:03:21.456422 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.456404 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:03:21.456471 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.456418 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:03:21.505465 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.505429 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:03:21.505465 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.505430 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:03:21.505689 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.505567 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:03:21.544650 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.544824 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.544824 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.544824 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.544824 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545010 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545010 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545010 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.544955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545133 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.545025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545133 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.545083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545133 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.545105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545357 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.545245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.545357 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.545290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.555888 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.555859 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:03:21.558276 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.558243 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:03:21.645848 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.645821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.645942 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.645858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.645942 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.645881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646058 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646101 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646101 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646220 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646220 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646329 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646329 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646329 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646329 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646515 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.646935 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.646905 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.647646 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.647545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.648015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.647987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.648766 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.648732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.649086 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.649058 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.649610 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.649542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.649929 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.649907 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.650551 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.650525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.650698 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.650680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.651156 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.651139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.651270 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.651252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.652378 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.652360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.710339 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.710316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.724307 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.724286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:21.754848 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.754817 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" event={"ID":"1ce2530e-5ad4-4eec-ad2c-58bca81cae53","Type":"ContainerStarted","Data":"f73dca5f7305308ff6d9aceb0999823883330f64a3a8c0f5b6cc50d889103cfb"} Oct 31 00:03:21.761858 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.761824 2576 generic.go:356] "Generic (PLEG): container finished" podID="a0a53051-3a82-499c-bd29-9ccf93a535bc" containerID="641dd2dfce8cd4a12e2effa36ef0b71e3922b52a336fb4ff4ec1c6b3a442fcea" exitCode=0 Oct 31 00:03:21.761963 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.761887 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67jm8" event={"ID":"a0a53051-3a82-499c-bd29-9ccf93a535bc","Type":"ContainerDied","Data":"641dd2dfce8cd4a12e2effa36ef0b71e3922b52a336fb4ff4ec1c6b3a442fcea"} Oct 31 00:03:21.797821 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.797794 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69b47fd846-94zxw"] Oct 31 00:03:21.800553 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.800531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.852592 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.852565 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vhzln\"" Oct 31 00:03:21.855392 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.855357 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5b339617vpk2d\"" Oct 31 00:03:21.866413 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.865848 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69b47fd846-94zxw"] Oct 31 00:03:21.949063 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khcqh\" (UniqueName: \"kubernetes.io/projected/2163faff-91f0-450a-aa4e-66336810549e-kube-api-access-khcqh\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2163faff-91f0-450a-aa4e-66336810549e-tmp\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949391 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-grpc-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949391 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949507 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.949555 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.950255 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.949984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2163faff-91f0-450a-aa4e-66336810549e-metrics-client-ca\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:21.953385 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.953344 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:03:21.953654 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.953468 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:03:21.953654 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.953494 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:03:21.953654 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.953517 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:03:21.953654 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:21.953566 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:03:22.010614 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.010564 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" podStartSLOduration=1.558477428 podStartE2EDuration="3.010549853s" totalImagesPullingTime="1.452072425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:21.877156712 +0000 UTC m=+54.866668245" watchObservedRunningTime="2026-10-31 00:03:22.010549853 +0000 UTC m=+55.000061381" Oct 31 00:03:22.050738 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2163faff-91f0-450a-aa4e-66336810549e-metrics-client-ca\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khcqh\" (UniqueName: \"kubernetes.io/projected/2163faff-91f0-450a-aa4e-66336810549e-kube-api-access-khcqh\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2163faff-91f0-450a-aa4e-66336810549e-tmp\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-grpc-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051222 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.050987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051608 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.051347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2163faff-91f0-450a-aa4e-66336810549e-tmp\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.051825 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.051639 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2163faff-91f0-450a-aa4e-66336810549e-metrics-client-ca\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.053670 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.053641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.053949 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.053929 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-grpc-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.054062 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.054016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.054271 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.054252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.055744 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.055721 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.055853 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.055828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.112875 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.112840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khcqh\" (UniqueName: \"kubernetes.io/projected/2163faff-91f0-450a-aa4e-66336810549e-kube-api-access-khcqh\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.123326 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.123301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:22.163061 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.163033 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:03:22.166087 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:22.166061 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6d3e4_f576_4666_815d_f3aea0c5dab1.slice/crio-f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430 WatchSource:0}: Error finding container f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430: Status 404 returned error can't find the container with id f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430 Oct 31 00:03:22.500308 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.500070 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69b47fd846-94zxw"] Oct 31 00:03:22.502848 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:22.502823 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2163faff_91f0_450a_aa4e_66336810549e.slice/crio-76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80 WatchSource:0}: Error finding container 76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80: Status 404 returned error can't find the container with id 76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80 Oct 31 00:03:22.670712 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.670690 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:22.767284 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.767246 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67jm8" event={"ID":"a0a53051-3a82-499c-bd29-9ccf93a535bc","Type":"ContainerStarted","Data":"0b7d934e7496b2a39d633a3ccd8c68f91a12d8859e18e0917aea6d3090d121ec"} Oct 31 00:03:22.767766 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.767321 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67jm8" event={"ID":"a0a53051-3a82-499c-bd29-9ccf93a535bc","Type":"ContainerStarted","Data":"121b7c3b788655fccd4d4c042dd321cac20fdf8618dec71325db087478a38f50"} Oct 31 00:03:22.769559 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.769532 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" event={"ID":"a61bcee6-5eb5-44b0-af5b-81bf788bf136","Type":"ContainerStarted","Data":"658db26d18825907ec0aff6f380329366ed718fa835f29872b0694e84b57a536"} Oct 31 00:03:22.769711 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.769597 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" event={"ID":"a61bcee6-5eb5-44b0-af5b-81bf788bf136","Type":"ContainerStarted","Data":"b57d04d6fc16a1273c14ad40e20bc8271a091b7e94e47f2188550e8983220e4e"} Oct 31 00:03:22.769711 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.769626 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" event={"ID":"a61bcee6-5eb5-44b0-af5b-81bf788bf136","Type":"ContainerStarted","Data":"cc15ad2aeb424e1a429ef4ec2bcf4e0e98d4c46f3363592e14a981aaaa2a7a14"} Oct 31 00:03:22.771308 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.771287 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80"} Oct 31 00:03:22.772590 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.772554 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430"} Oct 31 00:03:22.917018 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:22.916967 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-67jm8" podStartSLOduration=2.420946749 podStartE2EDuration="3.916954108s" totalImagesPullingTime="1.414417979s" totalInitContainerRuntime="81.58938ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:22.916490002 +0000 UTC m=+55.906001528" watchObservedRunningTime="2026-10-31 00:03:22.916954108 +0000 UTC m=+55.906465703" Oct 31 00:03:23.034223 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.034105 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" podStartSLOduration=2.146197175 podStartE2EDuration="4.034087448s" totalImagesPullingTime="1.887890273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:23.033490476 +0000 UTC m=+56.023002004" watchObservedRunningTime="2026-10-31 00:03:23.034087448 +0000 UTC m=+56.023598973" Oct 31 00:03:23.237801 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.237739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:23.439444 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.439365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:23.583083 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.583047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:23.729653 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.729624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:23.780663 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.780622 2576 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="af4412f13fe206bb249b210e92b982f1568dce2faa9567a05605ddce0c3ea805" exitCode=0 Oct 31 00:03:23.781105 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.780713 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"af4412f13fe206bb249b210e92b982f1568dce2faa9567a05605ddce0c3ea805"} Oct 31 00:03:23.805825 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.805788 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-st7jb"] Oct 31 00:03:23.809116 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.809093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:23.866252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.865282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/888e451d-b53f-487c-bd01-c90a6ad9bf8a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:23.866252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.865603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rm4\" (UniqueName: \"kubernetes.io/projected/888e451d-b53f-487c-bd01-c90a6ad9bf8a-kube-api-access-z9rm4\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:23.867640 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.867618 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-st7jb"] Oct 31 00:03:23.871173 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.871102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:23.890722 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.890697 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r2q95\"" Oct 31 00:03:23.890722 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.890712 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:03:23.966737 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.966639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rm4\" (UniqueName: \"kubernetes.io/projected/888e451d-b53f-487c-bd01-c90a6ad9bf8a-kube-api-access-z9rm4\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:23.966737 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.966708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/888e451d-b53f-487c-bd01-c90a6ad9bf8a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:23.969855 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:23.969820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/888e451d-b53f-487c-bd01-c90a6ad9bf8a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:24.018444 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.018401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:24.030558 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.030531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rm4\" (UniqueName: \"kubernetes.io/projected/888e451d-b53f-487c-bd01-c90a6ad9bf8a-kube-api-access-z9rm4\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:24.069865 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.069832 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86d6c7656b-2hqmr"] Oct 31 00:03:24.073455 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.073427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.123198 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.123145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:24.126001 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.125972 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5lv6p\"" Oct 31 00:03:24.126147 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.126003 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:03:24.126147 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.126034 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:03:24.126147 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.126117 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:03:24.126439 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.126347 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-75f6pg955fqte\"" Oct 31 00:03:24.126439 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.126414 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:03:24.133902 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.133827 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86d6c7656b-2hqmr"] Oct 31 00:03:24.168699 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.168659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-tls\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.168852 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.168723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-client-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.168852 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.168759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qsk8\" (UniqueName: \"kubernetes.io/projected/822e1679-db28-423e-91c0-0dd4cf714902-kube-api-access-4qsk8\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.168953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.168929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-client-certs\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.168987 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.168966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.169068 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.169049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/822e1679-db28-423e-91c0-0dd4cf714902-audit-log\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.169118 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.169102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-metrics-server-audit-profiles\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.269917 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.269875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-client-certs\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.270067 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.269929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.270067 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.269979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/822e1679-db28-423e-91c0-0dd4cf714902-audit-log\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.270067 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.270008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-metrics-server-audit-profiles\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.270169 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.270066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-tls\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.270169 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.270090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-client-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.270169 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.270114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsk8\" (UniqueName: \"kubernetes.io/projected/822e1679-db28-423e-91c0-0dd4cf714902-kube-api-access-4qsk8\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.271151 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.270405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/822e1679-db28-423e-91c0-0dd4cf714902-audit-log\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.271151 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.270742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.271151 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.271114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-metrics-server-audit-profiles\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.272646 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.272620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-client-certs\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.273003 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.272904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-tls\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.273003 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.272992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-client-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.331767 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.331728 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsk8\" (UniqueName: \"kubernetes.io/projected/822e1679-db28-423e-91c0-0dd4cf714902-kube-api-access-4qsk8\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.387762 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.387733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:24.497926 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.497889 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-st7jb"] Oct 31 00:03:24.501478 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:24.501445 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888e451d_b53f_487c_bd01_c90a6ad9bf8a.slice/crio-508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5 WatchSource:0}: Error finding container 508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5: Status 404 returned error can't find the container with id 508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5 Oct 31 00:03:24.766542 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.766453 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86d6c7656b-2hqmr"] Oct 31 00:03:24.769967 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:24.769935 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822e1679_db28_423e_91c0_0dd4cf714902.slice/crio-9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad WatchSource:0}: Error finding container 9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad: Status 404 returned error can't find the container with id 9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad Oct 31 00:03:24.786311 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.786279 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" event={"ID":"888e451d-b53f-487c-bd01-c90a6ad9bf8a","Type":"ContainerStarted","Data":"508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5"} Oct 31 00:03:24.787657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.787634 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" event={"ID":"822e1679-db28-423e-91c0-0dd4cf714902","Type":"ContainerStarted","Data":"9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad"} Oct 31 00:03:24.947126 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.943843 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs"] Oct 31 00:03:24.950298 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.950275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977170 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzcf\" (UniqueName: \"kubernetes.io/projected/504be407-0ef5-4f9f-b6f5-5674004659b8-kube-api-access-jqzcf\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977308 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-federate-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977308 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977308 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-metrics-client-ca\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977434 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-serving-certs-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:24.977529 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:24.977456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.002897 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.002875 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 00:03:25.003131 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.003114 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 00:03:25.003244 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.003226 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 00:03:25.003310 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.003265 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 00:03:25.003732 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.003714 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rs266\"" Oct 31 00:03:25.003855 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.003806 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 00:03:25.053229 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.053204 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 00:03:25.054763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.054738 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs"] Oct 31 00:03:25.078734 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.078702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.078817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-serving-certs-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.078920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.078959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzcf\" (UniqueName: \"kubernetes.io/projected/504be407-0ef5-4f9f-b6f5-5674004659b8-kube-api-access-jqzcf\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.078990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-federate-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.079038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.079077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-metrics-client-ca\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.079228 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.079114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.080254 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.080223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.081859 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.081833 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-serving-certs-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.082345 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.082277 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-metrics-client-ca\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.083619 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.083593 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.087981 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.087936 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-federate-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.090446 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.090405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.090700 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.090678 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.149395 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.149332 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzcf\" (UniqueName: \"kubernetes.io/projected/504be407-0ef5-4f9f-b6f5-5674004659b8-kube-api-access-jqzcf\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.297566 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.297479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:25.793840 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.793813 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs"] Oct 31 00:03:25.796397 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.796368 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"f7f9c8a3e53630b7ef6fbe7f67234ea605e31aa6a9130c659fba22201cd28a8d"} Oct 31 00:03:25.796494 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.796450 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"4a9633ccb86d398d8b997c246408f3ea88feea138851459c8f21144634c60b08"} Oct 31 00:03:25.796543 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.796494 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"f13d6325a628c31ef2fe506ff83724d713eea0c1735968264761eafb9bed65f3"} Oct 31 00:03:25.801736 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:25.800994 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"832ed388a17b39340d188b8181d4452941009d71554f1f583d1098db63de592e"} Oct 31 00:03:25.820394 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:25.820337 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504be407_0ef5_4f9f_b6f5_5674004659b8.slice/crio-03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567 WatchSource:0}: Error finding container 03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567: Status 404 returned error can't find the container with id 03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567 Oct 31 00:03:26.414192 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.414149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:26.615490 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.615398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:26.758820 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.758789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:26.810865 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.808026 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" event={"ID":"504be407-0ef5-4f9f-b6f5-5674004659b8","Type":"ContainerStarted","Data":"03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567"} Oct 31 00:03:26.817035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.816320 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"b4d99e53c1b3507b5390b0c152e4a3cb50c1e5ad558cfec72a5224c11edc0abf"} Oct 31 00:03:26.817035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.816385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"65c10d8767d72da3ae1bc0bf47ab2c6e1b9a7b9173f03d9b5d0f7f7f1ad7e978"} Oct 31 00:03:26.817035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.816424 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" event={"ID":"2163faff-91f0-450a-aa4e-66336810549e","Type":"ContainerStarted","Data":"82b3893ea29f3cd495f31d31cd822c3ef1711653dfdb2ee36fe19280fccadba2"} Oct 31 00:03:26.817035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.816493 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:26.819819 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.819789 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"a8358481b86a5b6e9d8debe9595ff71b9194544626b76205205f26503bb15a5c"} Oct 31 00:03:26.819924 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.819856 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"a48d17a0f476314c5e5b588c64530e0499722487853dd1315961d904902baa4a"} Oct 31 00:03:26.819924 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.819885 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"142322dff666eeb4d1d06367790c8a7f3ef69540504cb6fd1730ec8751cf635b"} Oct 31 00:03:26.819924 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.819912 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"d4f569c79edac59f760501c8bae071807aed5d82eed6e341d7b94de0e87d1649"} Oct 31 00:03:26.821424 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.821402 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" event={"ID":"888e451d-b53f-487c-bd01-c90a6ad9bf8a","Type":"ContainerStarted","Data":"93274bfb1afa7d5438ed82b4b90423ed273d81a29eba9ff0f02565ba89fd5070"} Oct 31 00:03:26.821622 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.821602 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:26.827858 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.827840 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:26.916793 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.916770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:26.939076 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:26.939028 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" podStartSLOduration=2.2439638 podStartE2EDuration="5.939013873s" totalImagesPullingTime="3.695050073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:26.938925065 +0000 UTC m=+59.928436607" watchObservedRunningTime="2026-10-31 00:03:26.939013873 +0000 UTC m=+59.928525401" Oct 31 00:03:27.055875 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.055812 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" podStartSLOduration=2.052636634 podStartE2EDuration="4.055791744s" totalImagesPullingTime="2.00315511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:27.054990489 +0000 UTC m=+60.044502018" watchObservedRunningTime="2026-10-31 00:03:27.055791744 +0000 UTC m=+60.045303276" Oct 31 00:03:27.122402 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.122359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:27.338404 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.338377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:27.509411 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.509375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:27.654714 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.654617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:27.739480 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.739447 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nlmmj"] Oct 31 00:03:27.743880 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.743841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.796036 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.796008 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:03:27.796225 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.796044 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:03:27.796225 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.796142 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-89h7l\"" Oct 31 00:03:27.801933 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.801897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdac28b2-b646-4bcd-99d5-2c617094ea07-tmp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.802150 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.801954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswxp\" (UniqueName: \"kubernetes.io/projected/fdac28b2-b646-4bcd-99d5-2c617094ea07-kube-api-access-nswxp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.803409 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.803385 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nlmmj"] Oct 31 00:03:27.804352 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.804290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:27.828383 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.828355 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerStarted","Data":"b3ee241e93217d77933eaa1b1b619d8e1030d9861889ab89eb85dfa5fc3cd38e"} Oct 31 00:03:27.829931 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.829908 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" event={"ID":"822e1679-db28-423e-91c0-0dd4cf714902","Type":"ContainerStarted","Data":"3ed518827efb8f7c4132e9420daa98bc72c4760d2dd5ff8821434c36c4f42ae9"} Oct 31 00:03:27.903909 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.903877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswxp\" (UniqueName: \"kubernetes.io/projected/fdac28b2-b646-4bcd-99d5-2c617094ea07-kube-api-access-nswxp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.904357 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.904334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdac28b2-b646-4bcd-99d5-2c617094ea07-tmp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.904778 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.904727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdac28b2-b646-4bcd-99d5-2c617094ea07-tmp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.965934 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.965914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswxp\" (UniqueName: \"kubernetes.io/projected/fdac28b2-b646-4bcd-99d5-2c617094ea07-kube-api-access-nswxp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:27.975950 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:27.975901 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" podStartSLOduration=1.8892049050000002 podStartE2EDuration="3.975887403s" totalImagesPullingTime="2.086682498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:27.974749718 +0000 UTC m=+60.964261246" watchObservedRunningTime="2026-10-31 00:03:27.975887403 +0000 UTC m=+60.965399001" Oct 31 00:03:28.056382 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.056234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:28.107806 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.107760 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.92290792 podStartE2EDuration="7.107744047s" totalImagesPullingTime="4.169555772s" totalInitContainerRuntime="15.280355ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:28.105994152 +0000 UTC m=+61.095505683" watchObservedRunningTime="2026-10-31 00:03:28.107744047 +0000 UTC m=+61.097255610" Oct 31 00:03:28.133748 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.133722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:28.160994 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.160931 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:03:28.166746 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.166718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207447 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207567 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207567 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207638 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207638 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207718 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207718 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207816 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207874 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207922 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207970 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.207970 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.207961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208035 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208066 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208096 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208127 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208156 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208211 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208252 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.208293 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.208251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.219249 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219222 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:03:28.219378 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219360 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:03:28.219634 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219609 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219669 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219670 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219692 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219714 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219748 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219756 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:03:28.219763 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219746 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:03:28.220088 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219821 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lb62x\"" Oct 31 00:03:28.220088 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219823 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:03:28.220088 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.219865 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-adcr02j2qj3ro\"" Oct 31 00:03:28.221117 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.221101 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:03:28.221307 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.221292 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:03:28.226294 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.226279 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:03:28.270333 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.270299 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:03:28.298657 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.298608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:28.308843 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.308813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.308960 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.308882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.308960 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.308908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.308960 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.308937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309109 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.308966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309159 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309297 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309297 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309386 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309386 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309386 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309517 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309517 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309517 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309517 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309517 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309912 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309912 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309912 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309912 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.309912 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.309842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.310279 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.310254 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.310499 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.310476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.310601 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.310577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.312288 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.311267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.312288 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.312200 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.312904 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.312876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.313385 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.313359 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.313979 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.313942 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.314540 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.314504 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.314640 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.314525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.314783 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.314757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.314954 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.314913 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.315041 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.314989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.315583 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.315550 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.315583 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.315567 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.315840 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.315818 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.316742 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.316727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.316858 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.316842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.375547 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.375523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.476673 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.476600 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nlmmj"] Oct 31 00:03:28.479892 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:28.479863 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdac28b2_b646_4bcd_99d5_2c617094ea07.slice/crio-a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4 WatchSource:0}: Error finding container a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4: Status 404 returned error can't find the container with id a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4 Oct 31 00:03:28.491678 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.491655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:28.745051 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.744982 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:28.837628 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.837591 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" event={"ID":"504be407-0ef5-4f9f-b6f5-5674004659b8","Type":"ContainerStarted","Data":"8b0a567297d060ab9551c323251ffa4e32e01860afe3143c88ab50b86dd46fb6"} Oct 31 00:03:28.838015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.837658 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" event={"ID":"504be407-0ef5-4f9f-b6f5-5674004659b8","Type":"ContainerStarted","Data":"f68ec01122e8cbe248c435e30e343b02e3fcf6250f77eacebb2d45266c3f77cf"} Oct 31 00:03:28.838015 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.837684 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" event={"ID":"504be407-0ef5-4f9f-b6f5-5674004659b8","Type":"ContainerStarted","Data":"c0141fb5b3f6d6b9290fa7b6172e505e8bc09f3a8d836efb9dc3f29639cb65ac"} Oct 31 00:03:28.838850 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.838823 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nlmmj" event={"ID":"fdac28b2-b646-4bcd-99d5-2c617094ea07","Type":"ContainerStarted","Data":"a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4"} Oct 31 00:03:28.923258 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.923229 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:03:28.946777 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.946742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:28.951078 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:28.951053 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5616844_000c_4b6c_8150_e7421eb3146c.slice/crio-d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb WatchSource:0}: Error finding container d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb: Status 404 returned error can't find the container with id d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb Oct 31 00:03:28.982663 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:28.982623 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" podStartSLOduration=2.9574001450000003 podStartE2EDuration="4.982607766s" totalImagesPullingTime="2.025207621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:28.982229474 +0000 UTC m=+61.971741001" watchObservedRunningTime="2026-10-31 00:03:28.982607766 +0000 UTC m=+61.972119294" Oct 31 00:03:29.251110 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:29.251016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:29.845112 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:29.845074 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" exitCode=0 Oct 31 00:03:29.845514 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:29.845163 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} Oct 31 00:03:29.845514 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:29.845264 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb"} Oct 31 00:03:30.569714 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:30.569677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:30.722787 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:30.722749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:30.883369 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:30.883279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:31.039552 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:31.039518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:32.448712 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.448676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:32.449197 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.448763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:32.549665 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.549531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:32.549665 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.549610 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:03:32.549665 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.549610 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:03:32.562741 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.562713 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:32.563515 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.563462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:32.601138 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.601108 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:03:32.636200 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.636160 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\"" Oct 31 00:03:32.645514 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.645486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:32.647368 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.647346 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\"" Oct 31 00:03:32.656725 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.656681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:32.661817 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.661547 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:03:32.680596 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.680567 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:32.837894 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.837863 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:32.941315 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.941285 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\"" Oct 31 00:03:32.950987 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:32.950960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:33.039859 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:33.039769 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-68spz"] Oct 31 00:03:33.085392 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:33.085362 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc7e49e_8890_47c1_a64f_25328161fde3.slice/crio-8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5 WatchSource:0}: Error finding container 8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5: Status 404 returned error can't find the container with id 8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5 Oct 31 00:03:33.091283 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:33.091160 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4slbb"] Oct 31 00:03:33.093531 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:33.093506 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725956a2_d668_4d42_989a_f5aeca0b2288.slice/crio-94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0 WatchSource:0}: Error finding container 94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0: Status 404 returned error can't find the container with id 94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0 Oct 31 00:03:33.328789 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:33.328756 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2kmrp"] Oct 31 00:03:33.331932 ip-10-0-143-53 kubenswrapper[2576]: W1031 00:03:33.331895 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f2d18c_7a3f_498f_b66e_b104cbab0e6c.slice/crio-17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182 WatchSource:0}: Error finding container 17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182: Status 404 returned error can't find the container with id 17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182 Oct 31 00:03:33.863883 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:33.863844 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2kmrp" event={"ID":"38f2d18c-7a3f-498f-b66e-b104cbab0e6c","Type":"ContainerStarted","Data":"17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182"} Oct 31 00:03:33.865134 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:33.865107 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-68spz" event={"ID":"1bc7e49e-8890-47c1-a64f-25328161fde3","Type":"ContainerStarted","Data":"8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5"} Oct 31 00:03:33.866413 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:33.866389 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4slbb" event={"ID":"725956a2-d668-4d42-989a-f5aeca0b2288","Type":"ContainerStarted","Data":"94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0"} Oct 31 00:03:34.878953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:34.877405 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} Oct 31 00:03:34.878953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:34.877538 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} Oct 31 00:03:34.878953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:34.877582 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} Oct 31 00:03:34.878953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:34.877623 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} Oct 31 00:03:34.878953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:34.877662 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} Oct 31 00:03:34.878953 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:34.877698 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerStarted","Data":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} Oct 31 00:03:35.066144 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:35.066091 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.009026863 podStartE2EDuration="7.066073111s" totalImagesPullingTime="4.041470719s" totalInitContainerRuntime="15.575529ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:35.064302263 +0000 UTC m=+68.053813795" watchObservedRunningTime="2026-10-31 00:03:35.066073111 +0000 UTC m=+68.055584638" Oct 31 00:03:35.826774 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:35.826741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:35.887218 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:35.887168 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4slbb" event={"ID":"725956a2-d668-4d42-989a-f5aeca0b2288","Type":"ContainerStarted","Data":"36db0c032c814de2cd07cf85d1fcccb561bb9bc4b8e8d10d43d5f5a1410ccf7e"} Oct 31 00:03:35.887659 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:35.887234 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4slbb" event={"ID":"725956a2-d668-4d42-989a-f5aeca0b2288","Type":"ContainerStarted","Data":"d7b66efdf7a6d2d35c187dda2808b8435d08b4d85f2087cef87e4762ed27b0b8"} Oct 31 00:03:36.009572 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:36.009520 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4slbb" podStartSLOduration=67.016503839 podStartE2EDuration="1m9.009502497s" totalImagesPullingTime="1.992998658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:36.009036811 +0000 UTC m=+68.998548334" watchObservedRunningTime="2026-10-31 00:03:36.009502497 +0000 UTC m=+68.999014027" Oct 31 00:03:36.024138 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:36.024037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:37.671798 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:37.671707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:37.898353 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:37.898318 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2kmrp" event={"ID":"38f2d18c-7a3f-498f-b66e-b104cbab0e6c","Type":"ContainerStarted","Data":"67440f4496ed0314e4d943292c30f3318bd5c7e3cfbecf42fa5d10748e29174a"} Oct 31 00:03:37.898510 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:37.898427 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:37.920851 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:37.920815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:38.492829 ip-10-0-143-53 kubenswrapper[2576]: I1031 00:03:38.492795 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:39.838741 ip-10-0-143-53 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:03:39.856878 ip-10-0-143-53 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:03:39.857118 ip-10-0-143-53 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:03:39.857210 ip-10-0-143-53 systemd[1]: kubelet.service: Consumed 4.882s CPU time. Oct 31 00:03:39.872819 ip-10-0-143-53 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:03:39.937414 ip-10-0-143-53 kubenswrapper[5943]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:03:39.937414 ip-10-0-143-53 kubenswrapper[5943]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:03:39.937414 ip-10-0-143-53 kubenswrapper[5943]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:03:39.937414 ip-10-0-143-53 kubenswrapper[5943]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:03:39.939515 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939435 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.939515 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939513 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.939515 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939517 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.939515 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939520 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939523 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939526 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939529 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939534 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939538 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939541 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939545 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939547 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939550 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939553 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939556 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939565 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939568 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939571 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939573 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939576 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939579 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939581 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939584 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.939644 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939586 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939589 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939592 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939594 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939597 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939600 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939603 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939605 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939608 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939610 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939613 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939615 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939618 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939620 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939622 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939625 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939627 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939630 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939633 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.940167 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939635 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939638 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939640 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939643 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939646 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939649 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939652 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939655 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939658 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939661 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939663 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939665 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939668 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939671 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939675 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939679 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939682 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939687 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939690 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.940708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939693 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939695 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939698 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939700 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939703 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939705 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939708 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939710 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939713 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939715 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939718 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939720 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939723 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939726 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939728 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939730 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939734 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939736 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939739 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939742 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.941254 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939744 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939747 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939749 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939751 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939754 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939756 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939759 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939761 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939765 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939767 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939771 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939774 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939776 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939780 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939782 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939785 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939787 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939790 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939792 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.941719 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939795 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939797 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939799 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939802 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939804 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.939807 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940696 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940705 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940708 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940710 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940719 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940722 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940726 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940731 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940736 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940739 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940742 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940744 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940747 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.942168 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940750 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940752 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940754 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940790 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940794 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940797 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940799 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940802 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940804 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940807 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940809 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940812 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940814 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940817 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940819 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940821 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940824 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940826 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940829 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940831 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.942633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940833 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940836 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940838 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940841 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940850 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940853 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940855 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940859 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940861 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940864 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940866 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940869 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940871 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940874 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940876 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940878 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940881 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940883 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940886 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940888 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.943105 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940891 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940894 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940902 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940905 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940907 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940910 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940913 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940915 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940918 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940920 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940928 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940931 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940933 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940935 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940938 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940941 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940943 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940951 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940954 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940957 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.943613 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940959 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940962 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940964 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940967 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940970 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940972 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940975 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940977 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940980 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940982 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940985 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940987 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940990 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940992 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940995 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.940997 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941000 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941003 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941006 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.944117 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941008 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941011 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941015 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941017 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941020 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941023 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941027 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941031 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941035 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941039 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941043 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941059 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941065 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941067 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941643 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941649 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941653 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941656 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941659 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941661 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.944642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941664 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941667 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941670 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941672 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941675 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941678 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941680 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941683 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941685 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941687 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941690 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941692 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941695 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941697 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941703 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941705 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941708 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941710 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941713 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.945285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941715 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941718 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941720 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941723 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941727 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941730 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941732 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941735 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941737 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941740 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941743 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941746 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941748 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941751 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941753 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941756 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941758 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941761 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941764 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941766 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.945745 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941768 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941779 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941782 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941786 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941790 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941792 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941795 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941800 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941803 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941807 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941811 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941814 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941818 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941823 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941826 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941830 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941834 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941839 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941843 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.946342 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941847 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941851 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941854 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941858 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941862 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941865 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941868 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941870 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941873 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941875 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941878 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941880 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941883 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941886 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941888 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941891 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941893 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941896 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941898 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.947116 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941900 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941905 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941907 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941910 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941913 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941915 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941918 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941921 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941923 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941926 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941929 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941932 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941935 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941937 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941940 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941944 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941947 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941951 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941953 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941955 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.947902 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941958 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941960 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.941963 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942058 5943 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942073 5943 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942080 5943 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942084 5943 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942088 5943 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942091 5943 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942095 5943 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942099 5943 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942102 5943 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942105 5943 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942110 5943 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942113 5943 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942116 5943 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942119 5943 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942122 5943 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942125 5943 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942127 5943 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942130 5943 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942134 5943 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942136 5943 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942139 5943 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:03:39.948800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942142 5943 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942149 5943 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942153 5943 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942157 5943 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942160 5943 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942164 5943 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942167 5943 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942170 5943 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942173 5943 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942189 5943 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942193 5943 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942197 5943 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942200 5943 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942203 5943 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942205 5943 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942208 5943 flags.go:64] FLAG: --enable-server="true" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942211 5943 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942216 5943 flags.go:64] FLAG: --event-burst="100" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942219 5943 flags.go:64] FLAG: --event-qps="50" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942221 5943 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942224 5943 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942229 5943 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942233 5943 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942236 5943 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942238 5943 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:03:39.949876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942241 5943 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942244 5943 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942247 5943 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942250 5943 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942253 5943 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942256 5943 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942259 5943 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942261 5943 flags.go:64] FLAG: --feature-gates="" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942266 5943 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942269 5943 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942274 5943 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942277 5943 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942280 5943 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942283 5943 flags.go:64] FLAG: --help="false" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942286 5943 flags.go:64] FLAG: --hostname-override="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942289 5943 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942291 5943 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942294 5943 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942297 5943 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942301 5943 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942303 5943 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942306 5943 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942309 5943 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:03:39.951017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942312 5943 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942314 5943 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942318 5943 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942320 5943 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942323 5943 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942327 5943 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942330 5943 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942333 5943 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942335 5943 flags.go:64] FLAG: --lock-file="" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942338 5943 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942341 5943 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942344 5943 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942349 5943 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942352 5943 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942354 5943 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942357 5943 flags.go:64] FLAG: --logging-format="text" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942360 5943 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942363 5943 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942366 5943 flags.go:64] FLAG: --manifest-url="" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942370 5943 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942375 5943 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942377 5943 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942381 5943 flags.go:64] FLAG: --max-pods="110" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942384 5943 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942387 5943 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:03:39.952093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942389 5943 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942392 5943 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942395 5943 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942398 5943 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942401 5943 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942408 5943 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942411 5943 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942413 5943 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942417 5943 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942419 5943 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942422 5943 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942425 5943 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942429 5943 flags.go:64] FLAG: --port="10250" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942432 5943 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942435 5943 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-08b8569304e5bba5d" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942438 5943 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942441 5943 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942444 5943 flags.go:64] FLAG: --register-node="true" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942446 5943 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942450 5943 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942453 5943 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942455 5943 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942458 5943 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942462 5943 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942465 5943 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942468 5943 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:03:39.953200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942472 5943 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942475 5943 flags.go:64] FLAG: --runonce="false" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942478 5943 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942481 5943 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942484 5943 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942487 5943 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942490 5943 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942493 5943 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942496 5943 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942499 5943 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942502 5943 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942504 5943 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942507 5943 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942510 5943 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942513 5943 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942516 5943 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942518 5943 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942524 5943 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942528 5943 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942530 5943 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942535 5943 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942538 5943 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942541 5943 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942543 5943 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942546 5943 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:03:39.954351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942549 5943 flags.go:64] FLAG: --v="2" Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942554 5943 flags.go:64] FLAG: --version="false" Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942558 5943 flags.go:64] FLAG: --vmodule="" Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942562 5943 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.942565 5943 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942706 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942710 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942715 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942718 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942721 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942724 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942726 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942729 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942731 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942734 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942736 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942739 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942741 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942744 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942746 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942749 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.955446 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942751 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942753 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942756 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942759 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942762 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942765 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942767 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942770 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942772 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942775 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942777 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942780 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942782 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942785 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942788 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942791 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942795 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942798 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942802 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.956361 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942805 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942808 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942811 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942813 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942816 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942819 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942821 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942824 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942826 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942828 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942831 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942833 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942836 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942838 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942841 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942843 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942846 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942849 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942851 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942854 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.956962 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942856 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942859 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942861 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942864 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942866 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942869 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942871 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942874 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942876 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942878 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942881 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942884 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942888 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942893 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942897 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942902 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942907 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942910 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942914 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942918 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.957581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942922 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942926 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942929 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942933 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942939 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942945 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942951 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942957 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942962 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942967 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942970 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942973 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942976 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942978 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942981 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942983 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942986 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942988 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942991 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.958410 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942994 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942996 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.942999 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943001 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943005 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943007 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943010 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943013 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943015 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943018 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943021 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943023 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.943050 5943 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943084 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943088 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943091 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943093 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.958928 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943096 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943098 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943101 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943103 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943105 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943108 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943112 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943115 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943117 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943120 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943122 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943125 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943127 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943130 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943132 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943135 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943137 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943140 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943142 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943145 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.959408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943148 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943151 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943153 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943158 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943162 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943165 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943168 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943171 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943187 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943190 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943193 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943195 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943199 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943201 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943204 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943206 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943208 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943211 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943217 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.959990 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943219 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943222 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943224 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943227 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943229 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943232 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943234 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943237 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943239 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943242 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943245 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943247 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943250 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943252 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943255 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943257 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943259 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943262 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943264 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943267 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.960604 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943270 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943272 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943275 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943277 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943280 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943282 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943284 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943287 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943289 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943292 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943294 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943298 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943302 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943306 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943309 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943311 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943314 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943316 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943319 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.961109 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943321 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943324 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943326 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943329 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943331 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943334 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943336 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943339 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943342 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943344 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943347 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943349 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943352 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943355 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943357 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943360 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943362 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943365 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943367 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943370 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.961587 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943372 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943375 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943377 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.943380 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.949376 5943 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.949395 5943 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.949427 5943 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.949436 5943 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949524 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949533 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949538 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949543 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949548 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949553 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949558 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949562 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949567 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949571 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949576 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.962065 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949581 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949585 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949591 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949596 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949601 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949605 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949609 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949615 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949619 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949624 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949629 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949634 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949638 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949643 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949648 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949653 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949657 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949669 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949674 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949679 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.962540 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949684 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949689 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949693 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949698 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949702 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949707 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949711 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949716 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949721 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949725 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949730 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949735 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949739 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949743 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949748 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949753 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949757 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949762 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949768 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949781 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.963050 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949786 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949790 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949795 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949801 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949806 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949810 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949815 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949820 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949826 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949832 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949839 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949844 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949849 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949855 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949860 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949864 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949869 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949873 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949878 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.963867 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949882 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949886 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949890 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949895 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949899 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949903 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949910 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949917 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949923 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949937 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949942 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949946 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949951 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949956 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949960 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949965 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949970 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949975 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949980 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.964610 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949984 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949988 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949993 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.949997 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950002 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950007 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950012 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950016 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950021 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950025 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950030 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950034 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950038 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950043 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950048 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950053 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.965104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950058 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.950101 5943 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950343 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950354 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950358 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950363 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950368 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950381 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950386 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950391 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950396 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950401 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950405 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950410 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950414 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950419 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.965526 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950423 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950428 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950432 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950437 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950441 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950446 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950451 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950455 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950460 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950464 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950469 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950474 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950478 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950483 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950487 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950492 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950496 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950501 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950506 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950511 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.965924 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950515 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950520 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950524 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950529 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950536 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950541 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950546 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950553 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950559 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950565 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950570 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950575 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950579 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950584 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950589 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950593 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950598 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950602 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950607 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.966496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950611 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950616 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950621 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950625 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950630 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950635 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950640 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950645 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950650 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950654 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950659 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950663 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950668 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950672 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950677 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950681 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950686 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950691 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950696 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950701 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.966953 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950705 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950711 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950721 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950727 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950732 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950737 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950741 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950746 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950750 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950755 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950759 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950764 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950769 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950774 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950779 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950783 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950788 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950792 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950796 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.967565 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950801 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950806 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950810 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950815 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950819 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950824 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950828 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950833 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950837 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950842 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950846 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950851 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950856 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950861 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.950900 5943 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950957 5943 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:03:39.968106 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950966 5943 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950971 5943 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950976 5943 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950981 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950986 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950990 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950995 5943 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.950999 5943 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951004 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951008 5943 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951013 5943 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951017 5943 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951021 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951026 5943 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951032 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951037 5943 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951043 5943 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951047 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951052 5943 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951057 5943 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:03:39.968570 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951062 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951067 5943 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951071 5943 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951076 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951081 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951085 5943 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951090 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951095 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951100 5943 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951105 5943 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951110 5943 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951115 5943 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951120 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951124 5943 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951129 5943 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951133 5943 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951138 5943 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951142 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951147 5943 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951152 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:03:39.969274 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951156 5943 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951161 5943 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951165 5943 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951169 5943 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951192 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951197 5943 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951203 5943 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951209 5943 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951214 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951219 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951224 5943 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951228 5943 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951233 5943 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951237 5943 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951241 5943 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951247 5943 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951251 5943 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951256 5943 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951261 5943 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:03:39.969784 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951265 5943 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951270 5943 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951275 5943 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951281 5943 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951316 5943 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951324 5943 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951330 5943 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951335 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951340 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951345 5943 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951350 5943 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951354 5943 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951359 5943 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951364 5943 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951368 5943 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951373 5943 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951377 5943 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951382 5943 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951386 5943 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951393 5943 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:03:39.970496 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951398 5943 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951402 5943 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951407 5943 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951413 5943 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951417 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951422 5943 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951427 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951431 5943 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951436 5943 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951443 5943 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951449 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951455 5943 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951460 5943 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951465 5943 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951471 5943 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951476 5943 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951481 5943 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951486 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951492 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:03:39.971285 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951497 5943 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951501 5943 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951506 5943 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951511 5943 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951515 5943 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951520 5943 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:03:39.951524 5943 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.951605 5943 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.953165 5943 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.953256 5943 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.953538 5943 server.go:1014] "Starting client certificate rotation" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.953662 5943 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.953702 5943 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:57:27 +0000 UTC" deadline="2028-03-29 01:19:00.166081782 +0000 UTC" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.953729 5943 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12361h15m20.212355083s" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.954304 5943 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.956066 5943 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:03:39.971925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.957499 5943 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:03:39.972471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.959483 5943 log.go:25] "Validated CRI v1 image API" Oct 31 00:03:39.972471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.960284 5943 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:03:39.972471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.963032 5943 fs.go:135] Filesystem UUIDs: map[3389db9f-25d5-4298-b0e5-c026e0ba6bbc:/dev/nvme0n1p4 680fe705-c04c-4153-b0db-526a208b930c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 00:03:39.972586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.963047 5943 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182/userdata/shm major:0 minor:479 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b84b8341de2e3c5f36ef8877fb30f161950dac97c7ea446e6119147d997c05b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b84b8341de2e3c5f36ef8877fb30f161950dac97c7ea446e6119147d997c05b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5/userdata/shm major:0 minor:474 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0/userdata/shm major:0 minor:476 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4/userdata/shm major:0 minor:449 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb/userdata/shm major:0 minor:468 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2/volumes/kubernetes.io~projected/kube-api-access-pftp5:{mountpoint:/var/lib/kubelet/pods/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2/volumes/kubernetes.io~projected/kube-api-access-pftp5 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bc7e49e-8890-47c1-a64f-25328161fde3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1bc7e49e-8890-47c1-a64f-25328161fde3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~projected/kube-api-access-x978l:{mountpoint:/var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~projected/kube-api-access-x978l major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~projected/kube-api-access-bg4pb:{mountpoint:/var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~projected/kube-api-access-bg4pb major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~projected/kube-api-access-khcqh:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~projected/kube-api-access-khcqh major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/386772ab-964a-485d-9489-75e74e1fe1de/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/386772ab-964a-485d-9489-75e74e1fe1de/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38f2d18c-7a3f-498f-b66e-b104cbab0e6c/volumes/kubernetes.io~projected/kube-api-access-dkwr7:{mountpoint:/var/lib/kubelet/pods/38f2d18c-7a3f-498f-b66e-b104cbab0e6c/volumes/kubernetes.io~projected/kube-api-access-dkwr7 major:0 minor:478 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/499771b6-6b5b-4ad2-80e9-85f61d4d48e4/volumes/kubernetes.io~projected/kube-api-access-5gzn2:{mountpoint:/var/lib/kubelet/pods/499771b6-6b5b-4ad2-80e9-85f61d4d48e4/volumes/kubernetes.io~projected/kube-api-access-5gzn2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~projected/kube-api-access-jqzcf:{mountpoint:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~projected/kube-api-access-jqzcf major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e16f2f0-c342-414d-a8c8-5f68528c78b8/volumes/kubernetes.io~projected/kube-api-access-ljjm8:{mountpoint:/var/lib/kubelet/pods/5e16f2f0-c342-414d-a8c8-5f68528c78b8/volumes/kubernetes.io~projected/kube-api-access-ljjm8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e16f2f0-c342-414d-a8c8-5f68528c78b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5e16f2f0-c342-414d-a8c8-5f68528c78b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712ccf72-9a61-465b-a2df-495354882144/volumes/kubernetes.io~projected/kube-api-access-mgsx7:{mountpoint:/var/lib/kubelet/pods/712ccf72-9a61-465b-a2df-495354882144/volumes/kubernetes.io~projected/kube-api-access-mgsx7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/725956a2-d668-4d42-989a-f5aeca0b2288/volumes/kubernetes.io~projected/kube-api-access-74p4g:{mountpoint:/var/lib/kubelet/pods/725956a2-d668-4d42-989a-f5aeca0b2288/volumes/kubernetes.io~projected/kube-api-access-74p4g major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/725956a2-d668-4d42-989a-f5aeca0b2288/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/725956a2-d668-4d42-989a-f5aeca0b2288/volumes/kubernetes.io~secret/metrics-certs major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c04250d-a6cd-48e4-97b7-2959641907a7/volumes/kubernetes.io~projected/kube-api-access-99cl7:{mountpoint:/var/lib/kubelet/pods/7c04250d-a6cd-48e4-97b7-2959641907a7/volumes/kubernetes.io~projected/kube-api-access-99cl7 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~projected/kube-api-access-4qsk8:{mountpoint:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~projected/kube-api-access-4qsk8 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/888e451d-b53f-487c-bd01-c90a6ad9bf8a/volumes/kubernetes.io~projected/kube-api-access-z9rm4:{mountpoint:/var/lib/kubelet/pods/888e451d-b53f-487c-bd01-c90a6ad9bf8a/volumes/kubernetes.io~projected/kube-api-access-z9rm4 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/888e451d-b53f-487c-bd01-c90a6ad9bf8a/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/888e451d-b53f-487c-bd01-c90a6ad9bf8a/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/955eeb9d-5b8c-473a-b0cf-69d6672353bf/volumes/kubernetes.io~projected/kube-api-access-db6fr:{mountpoint:/var/lib/kubelet/pods/955eeb9d-5b8c-473a-b0cf-69d6672353bf/volumes/kubernetes.io~projected/kube-api-access-db6fr major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/955eeb9d-5b8c-473a-b0cf-69d6672353bf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/955eeb9d-5b8c-473a-b0cf-69d6672353bf/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b5cca15-e32c-4fb2-8487-ace277115471/volumes/kubernetes.io~projected/kube-api-access-wwrl9:{mountpoint:/var/lib/kubelet/pods/9b5cca15-e32c-4fb2-8487-ace277115471/volumes/kubernetes.io~projected/kube-api-access-wwrl9 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e406f6b-223a-4073-914b-74b5226b09c4/volumes/kubernetes.io~projected/kube-api-access-mpvtj:{mountpoint:/var/lib/kubelet/pods/9e406f6b-223a-4073-914b-74b5226b09c4/volumes/kubernetes.io~projected/kube-api-access-mpvtj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~projected/kube-api-access-5hlbw:{mountpoint:/var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~projected/kube-api-access-5hlbw major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~projected/kube-api-access-jztr5:{mountpoint:/var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~projected/kube-api-access-jztr5 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/config-reloader/2 major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/prometheus/15 major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~projected/kube-api-access-jwbqw:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~projected/kube-api-access-jwbqw major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~projected/tls-assets major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/config major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/web-config major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/alertmanager/9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/config-reloader/6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~empty-dir/config-out major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~projected/kube-api-access-zv6bg:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~projected/kube-api-access-zv6bg major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~projected/tls-assets major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/config-volume major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/web-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volumes/kubernetes.io~projected/kube-api-access-jb59b:{mountpoint:/var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volumes/kubernetes.io~projected/kube-api-access-jb59b major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6dc42da-f207-4a5c-a481-6d5a65363802/volumes/kubernetes.io~projected/kube-api-access-bqh9g:{mountpoint:/var/lib/kubelet/pods/f6dc42da-f207-4a5c-a481-6d5a65363802/volumes/kubernetes.io~projected/kube-api-access-bqh9g major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6dc42da-f207-4a5c-a481-6d5a65363802/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f6dc42da-f207-4a5c-a481-6d5a65363802/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdac28b2-b646-4bcd-99d5-2c617094ea07/volumes/kubernetes.io~projected/kube-api-access-nswxp:{mountpoint:/var/lib/kubelet/pods/fdac28b2-b646-4bcd-99d5-2c617094ea07/volumes/kubernetes.io~projected/kube-api-access-nswxp major:0 minor:447 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5cc479a369bfad51cd21f54a9af72e79aaa427c84dcf36079780b3042c747d16/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a9002d09b87a55ef45a09385a98dc5c856307e1c930dc515ce20ff1bb7aa55f2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5b7c0194453c9036b11bafcaeec2698fbd4e9ffd96bbf8e627a22b04b14c89a0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/5d0d9e96f68a154b35eaf02bb7bd2be5ce27ffb15a2674870912e2f654890f77/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/93bd1ec828ac05ac30211c9feec1a9a007d71cdc60a50f90611a858e2248ea19/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4667fdee191468a538b3fe484f8d8c5d0eb985869f5b618cdff9170aa736b5e2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c303dbe200cce2d839f88b96a4729b540f1c88dcfedbcd3db65e9f2cfb76266d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cce564737d624b73637055d732da3e70e6d72d3b94c6f38dfa212dffdad4a951/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/982b09321d3f064e3b27967bb0277d90769d8c747671cdee499fc2fc5d96fdbf/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/933505e194ebbf818a3699a6bf3be2fb1770cb2961fe5c1867b523d87eb4e1b2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/e3d93c2846204130203e17a3febe2bf67b8f69ceb554405548683c1379933dda/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/7fb33144194efbe120232c2565d6911b31c960608bc901f886f14cbcf65277cc/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/5ca6789a7024a1a31ad88462c569953aab876edcd7380078c837e6e09da40bdc/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/78b648203e01e1d44ca7bc835aa3f2d490388b1772b5ed37810ca30be1df3d4b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/352d43809aacf45ac872bcdb97591befd0532684a7a6fde85bf2303a983f0a65/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8ebfa5da68751fe8b414d83bb2087fff7fe6404edc43950331e9596db34ad545/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7d21dc60d1565f969cb87f762cdcc22ec27f4bbd3a6982a8884e42dcc6e86bb4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0de000a524986daa4a411cc171e84fd87f0dcead190c10e9cb5e80df0719c43c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4e6159e1261626606e440862b3a8648a5695db0581faf7c3db0d9ddca28075c1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/691770de5e7ce9c01d5522d4a848958178b5b4427eb183b3f4a8f985081902ac/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7d12c23c072987a7152a58928e0dcd2cd7c570fa46e35270494952584205b030/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7794c93916e9f6da44d573ceccd5d0ad1f492b55565e205622f12912ab7c3b50/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/15d3d0451d2f862b1f2cf36ba8f800fe651d07b5ca75de79006da9d8ff9fbd9d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/827f2d30b6b1e19a6b112507cc769dec1682fa028c6c00886d20ecbbf9830fd5/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/841c3c3de17a538a4fadbcf682bf4725f333bf56783f503d98c1e5fdf1487d14/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/6dfbc5fd1e9f77bdc227008a04d6df6dc5d1751d9fc3782529ad2ee8a07bade6/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/19e8686c4d0a9c52412e4df9801011a9b8fcc7168e163a5c83d4e579e53dd726/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e09c629d6a099f060b97c47c848f461a09f86eb1ce67c9e893ce2104dc95bd15/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/c53bfb192c33bebd4e617396f19539f97bc107fda0227a760a143fc64708c683/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/ca77a5c60544947b7a1d651be38f5614cccc36321b1f32f793bb1146263a264a/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/bb24db71e68f9bd3eb5728f30713a34c28985336ed080fb5ed9ba304a01ccfd4/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/223ef04db81bb34bd1ccdec7f1b7bedf3b34773484d423fda4f2e64efa0c6129/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/b59fd45c28a2ee3cb3704b97bc980fa515a7bbed5075580cf7bd5231fe99f7f4/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/a683debb7b93b678a387e02951431246ac9763c167640958be62a1cf5ce7c706/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/da933a182a38fe3506de3452b4a0f4f544b115d39775f7de6808d1059c9f7dff/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/10ea65d8898053a209c3a683db521a1e8248ae8dc7c18bd8be1a137ec8f50dc9/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/bad46ec462f8e45e48662889f99c4921a32d0a9f6ddfe2ed1f83ff6d0854ae4e/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/e419407e13a72de7fe4c7e9c88f152a71b485d4347721310a23a181107763610/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/466308733ab74a881a4f242e9decb932e28cb1662a900bc5059950187bcc1533/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/05cc7757d55d25363bd7fc5d7a17cc093d4c0c8ad9a4c5686799b6995c60e203/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/28747d9a61d96782df6ce8e1a3ea92819388f90254bbc9ac5b18dfa98312898b/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/0bd19d79eddcbf6c6406ccfd1940d649e7e75429ea1250a0616f04891dd19ff1/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/7ab002ecc756438646cf8a7f7915243c9febaf71b4226d6a23c03b373a49311f/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/b3c0e69f745c6bde7966991c4059eb2f7c4980ec6f35b245818144dbd82b9d5d/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/ea786b1669ebc3358546e0c71b6f00b3d0d55108a80a96e04ba15ce71491e9a0/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/319b6d9a04616a1cfd2c1c69953ae912f0a289fbdf2a366517ede7f9e0cca544/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/12ff656c682421aff054ef38ce2ec7a4d5f5ecf8f4421a2c25e63939aa85091b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/67a0d8447459cd6ca7d7ae9ddc4cc2a7532e74af9597d5104ab42b0de3e03d49/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/c13b28bc4db04501e57aaeb713526be38ebf666c4b5bb34cc9761ebaf30fe08a/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/e5f782cb21e6f12fce70324a4ca1008cd49fe8d8e30acf21297ac4957a21f45b/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/e4c6cfcfa08c78dead9936489f93335535f1a60e13d9ceef075c130f2a0cc1ff/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/bfe7e7e4569c95c48f0e77983fc226d7a6ccb5c7cae105a1efd58741d7c9345b/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/45209fddbeb56bc09aca2f2381bf3b456b43c088e0558a85391130d1b20f4169/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/1861e6cd8ab71d2559daf86d9882a8cce17855a064f8954b1d19289c284926be/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/836de6486f73660cd0afc36a2b4944c69ab6eb0faca0dde9326f646d7073c13b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/0ae1305664283c7a49d014953f6544c17b63708285d1f2e4500cdf4fcc13f8b4/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/77fe557aa2d535c358b8d0529a54f562698b808cbc532735e870d39e501d8ac4/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/bc9069fc0be1bcbda2aad2b789ec9a4dc6920c1b77464362f6661375df85ba25/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/daeb5b9bd33f4ec631750e62a86d0d60d9881666e917677501f05834427242ca/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/84574bf0485757aabb7939d4c138388d9cffd4e1420315cde01e0ff752b6a7ed/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/dd67a6e3710c1ae83d949c5b309fd07a0bf706e6fa22aa3e94016e822f3beea3/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/fc87e98a4090bb8011b3f40d4c8542d3bebf1620fa9ba86cf52f1780b7a6a5b4/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/87ebc13b14f29b23f62d85d52da0615bf98201c4b7a1704851f124d3709a6613/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/dd91d987a2b11b8a9d685af17c23cee5aca956ab33f2af74d4518c21162b82e0/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/7fc030597515b4e7c4fe3ae2f2a6c40500e96f600e62bccd4fe49aebfea1b650/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/54b6f9ea5c4faa6aa61e1311d2f9ca253d43cc31894d595a3c48067c3a42dd40/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/a0d06a0e963db2003ede806e3953f9966477dc58480be4ec7c9ee671c99c9e2d/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/728a75f55f0bbd1be2ee81834cf0366a5d409b582a454dde39c13fd20e106e47/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/6fbe454c078725bdf9994bdad7789aca8c592d787976f7618e66d3f1887a39f9/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/7e20ae93b2309d0d30c64c2ed236095569d83bf89296920bcd3554ca9fe715b7/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/fec90e6f1756e4747c6300f570d751587f0f55249c356a31d627d5355d2638d8/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/16c8662c33526973f4cad5b1aaebca8858dd1f03c6ec16f617c5666b86cc5ee8/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/580533ac55cf180c79d6057b4b1f89d6aec217a39d7d5ab86bb78250a5cc7c14/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/4d6f7c37a2e6d1cebb80359478c4543cacfcd536366e396e6f343433aa1a1e00/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/a9b2b6ede7e1190ce12d74fb8370ecf89492859f5b880da73503e0cdfe8eede3/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/415fb5ca1f453e4abbab2792c594f592f52e95d53f1b1c67e53617fb44c78bca/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/5e597c875e7f1db5b820236b6a22785adb2fba7dee596ba3b6a103625b6741bc/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-531:{mountpoint:/var/lib/containers/storage/overlay/7a8b1e9655d8838f7e65e13992d5b23b4372710f79e71cdf252ff7c568467d75/merged major:0 minor:531 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/f11e7733ff3057b93ade819bcf3cb2bb4d8e279954cb3ce600d32ac3dc310876/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d6d767106362c97be73fee80cc3c6ab8c482d54e7644947c9583886feb38fe60/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1cfc9ba86aea9219250b41162ba1f6ed0799d94d77bffe1b3a1278794a56b2fa/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6132dccf2edae5396dcb01f2d903829d8e3d2d376277308cbc0da586a6498e92/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/19839f410d7a95df8de7671fd2032b1b8f7d69dfafd6be45227f6eb30f8dee88/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/25763d33fced9984361a2457347ae9ae2577dadf0e8ad8a22e7ac8c1e4d732ec/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/fdf4c14f0b7a448ac4eb445506a97a9cec89fa524917781cfb9368c0bfad751f/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/442c05062bbcb780e477cafcaa9465bfbc279ecdd6e0576850c3f74c487a98cf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0917c76c131795c76b228a011600bbfd1ddedd8f5e6d9ff81c974a477de93374/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/21a3b0790ba428fd0197961caf4393ea60ca33dd0a710435c81a955f7a025196/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:03:39.973520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.973002 5943 manager.go:217] Machine: {Timestamp:2026-10-31 00:03:39.972514804 +0000 UTC m=+0.065804983 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3142619 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a96320906f023dbf766023fc04ff6 SystemUUID:ec2a9632-0906-f023-dbf7-66023fc04ff6 BootID:84ce9d10-97a6-4871-86ff-ae7e2db8654a Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/712ccf72-9a61-465b-a2df-495354882144/volumes/kubernetes.io~projected/kube-api-access-mgsx7 DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/955eeb9d-5b8c-473a-b0cf-69d6672353bf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdac28b2-b646-4bcd-99d5-2c617094ea07/volumes/kubernetes.io~projected/kube-api-access-nswxp DeviceMajor:0 DeviceMinor:447 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb/userdata/shm DeviceMajor:0 DeviceMinor:468 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:456 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:455 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e406f6b-223a-4073-914b-74b5226b09c4/volumes/kubernetes.io~projected/kube-api-access-mpvtj DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68593e02ad90c137d99508cd4ded0704a2c3d39f23e58ff7e798942a00773b46/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:456 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f6dc42da-f207-4a5c-a481-6d5a65363802/volumes/kubernetes.io~projected/kube-api-access-bqh9g DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~projected/kube-api-access-5hlbw DeviceMajor:0 DeviceMinor:257 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~projected/kube-api-access-jqzcf DeviceMajor:0 DeviceMinor:373 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~projected/kube-api-access-jwbqw DeviceMajor:0 DeviceMinor:465 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:365 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:368 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03b3bc3d74269a37a2951a96d47ed1e51949e90458121e5d886e322205b16567/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:459 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82793ac4777200b898a0151c61589f4a5e5adf0fa60235e80944d7ca7afc1c54/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:256 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c26d3123e6e818faac8c54ad1fb87adc06653fea46a9c58ce97293cdddfbac5/userdata/shm DeviceMajor:0 DeviceMinor:474 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~projected/kube-api-access-x978l DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a531611de0538d9aa619b3b1c7ef3bb89017d762fda09b7939a83236cce48981/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a0a53051-3a82-499c-bd29-9ccf93a535bc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/888e451d-b53f-487c-bd01-c90a6ad9bf8a/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:338 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:327 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:456 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/725956a2-d668-4d42-989a-f5aeca0b2288/volumes/kubernetes.io~projected/kube-api-access-74p4g DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:255 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:342 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bc7e49e-8890-47c1-a64f-25328161fde3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:472 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/091c03f7646e8ec8cc2084a202b95882a9b360b3a45afa5c766fbde3ec4ca2fb/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9152c24dfdcd8e27b8d056bde1cb3347936b55e1a6640de964f23406d6ba59e1/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:344 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/38f2d18c-7a3f-498f-b66e-b104cbab0e6c/volumes/kubernetes.io~projected/kube-api-access-dkwr7 DeviceMajor:0 DeviceMinor:478 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1cbf2b1a-032c-4047-81db-998713cc8927/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:343 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:285 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a345df81cd65ad10579bd449d4870c5ab399f5c22f0752d8d01a85a6f6f854b4/userdata/shm DeviceMajor:0 DeviceMinor:449 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-531 DeviceMajor:0 DeviceMinor:531 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/386772ab-964a-485d-9489-75e74e1fe1de/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:290 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2/volumes/kubernetes.io~projected/kube-api-access-pftp5 DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae9e6c2b0086ce34d11933d6088a30feb32a643b01ce3255f59148a964e77b29/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f6e15c17bb7bae1249a56321d151b6dae98bb04ce2ebb4d02459d61418c5c2f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:457 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:322 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80cfcb417143dba324d5626a70139cae01cec5a50ea3be4897b2d16b04c0c9e4/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:458 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94ada3ee2f6066a16851ad7edda13e527991e68f85d2e24a55524690bf312aa0/userdata/shm DeviceMajor:0 DeviceMinor:476 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17a062b13f9c410c7ca523b9b961bdeb791a4432dd0b9192de62515f52135182/userdata/shm DeviceMajor:0 DeviceMinor:479 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:456 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e12670ed76bb7fafeae44e6a2bbb1ca183f5e011b78adf63e376d39e7d81cfa6/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76c13e98e30455f624fd440c3cb641ef60ecf018ce90a2b8d79e34fcb1bd3f80/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:461 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:462 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11ecaffa4353ec47519f246cf4dabee503cf4ab7609a59cc511041fad6ed2544/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~projected/kube-api-access-jztr5 DeviceMajor:0 DeviceMinor:258 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:291 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~projected/kube-api-access-khcqh DeviceMajor:0 DeviceMinor:331 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:367 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c04250d-a6cd-48e4-97b7-2959641907a7/volumes/kubernetes.io~projected/kube-api-access-99cl7 DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volumes/kubernetes.io~projected/kube-api-access-jb59b DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:326 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/508720edcd1278137d24b625a05672a573c098ef2064a21bf9137793b1e607d5/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/725956a2-d668-4d42-989a-f5aeca0b2288/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:473 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:454 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/504be407-0ef5-4f9f-b6f5-5674004659b8/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:366 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:464 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/888e451d-b53f-487c-bd01-c90a6ad9bf8a/volumes/kubernetes.io~projected/kube-api-access-z9rm4 DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0e39738422adcea28e808384decb28bd3d314ef40fafc4bd2e7005b1179da16/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69ab5da55b9cb6e9b0f53b2bb0c02f8c36da34e12b4647ec32fd2363167e1f85/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/955eeb9d-5b8c-473a-b0cf-69d6672353bf/volumes/kubernetes.io~projected/kube-api-access-db6fr DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:292 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/822e1679-db28-423e-91c0-0dd4cf714902/volumes/kubernetes.io~projected/kube-api-access-4qsk8 DeviceMajor:0 DeviceMinor:345 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b84b8341de2e3c5f36ef8877fb30f161950dac97c7ea446e6119147d997c05b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a61bcee6-5eb5-44b0-af5b-81bf788bf136/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e743ea0bc432139318288d228b36d70329311341115e5cb7ea37ce0a333fad5/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes/kubernetes.io~projected/kube-api-access-zv6bg DeviceMajor:0 DeviceMinor:293 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:285 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9b5cca15-e32c-4fb2-8487-ace277115471/volumes/kubernetes.io~projected/kube-api-access-wwrl9 DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e16f2f0-c342-414d-a8c8-5f68528c78b8/volumes/kubernetes.io~projected/kube-api-access-ljjm8 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce2530e-5ad4-4eec-ad2c-58bca81cae53/volumes/kubernetes.io~projected/kube-api-access-bg4pb DeviceMajor:0 DeviceMinor:252 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/499771b6-6b5b-4ad2-80e9-85f61d4d48e4/volumes/kubernetes.io~projected/kube-api-access-5gzn2 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e16f2f0-c342-414d-a8c8-5f68528c78b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f2a68b12b08d46bee2db0edb8dda26acd86e259d61da3791b65c1d158904bad/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:460 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6dc42da-f207-4a5c-a481-6d5a65363802/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2163faff-91f0-450a-aa4e-66336810549e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:463 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03b3bc3d74269a3 MacAddress:be:a5:77:95:09:c8 Speed:10000 Mtu:8901} {Name:11ecaffa4353ec4 MacAddress:86:65:b4:22:05:9c Speed:10000 Mtu:8901} {Name:17a062b13f9c410 MacAddress:be:e5:b6:76:9a:9b Speed:10000 Mtu:8901} {Name:2e743ea0bc43213 MacAddress:de:42:0f:cf:cc:54 Speed:10000 Mtu:8901} {Name:508720edcd12781 MacAddress:b2:cf:9c:cb:81:3f Speed:10000 Mtu:8901} {Name:69ab5da55b9cb6e MacAddress:16:90:2f:bc:68:59 Speed:10000 Mtu:8901} {Name:76c13e98e30455f MacAddress:66:f0:36:08:55:c7 Speed:10000 Mtu:8901} {Name:8c26d3123e6e818 MacAddress:4e:b3:b3:6e:38:c0 Speed:10000 Mtu:8901} {Name:9152c24dfdcd8e2 MacAddress:da:1c:ab:e3:16:f7 Speed:10000 Mtu:8901} {Name:94ada3ee2f6066a MacAddress:66:b8:82:a6:fc:4a Speed:10000 Mtu:8901} {Name:9f2a68b12b08d46 MacAddress:02:d4:a8:b9:10:7a Speed:10000 Mtu:8901} {Name:a345df81cd65ad1 MacAddress:7a:63:ab:a8:7b:b9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:48:97:9a:19:65 Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:b1:58:3e:27:9d Speed:0 Mtu:8901} {Name:d9aa1edab3625a3 MacAddress:1e:29:9e:a0:9f:9e Speed:10000 Mtu:8901} {Name:e12670ed76bb7fa MacAddress:7e:9f:00:f6:09:70 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:48:97:9a:19:65 Speed:0 Mtu:9001} {Name:f0ea0df8a8e3849 MacAddress:1e:96:c5:d8:d8:32 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7a:d3:49:ec:d8:6b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:00:1d:67:a4:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:03:39.973787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.973527 5943 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:03:39.973787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.973578 5943 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:03:39.973887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.973864 5943 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:03:39.974028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.973887 5943 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-53.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:03:39.974073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974030 5943 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:03:39.974073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974037 5943 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:03:39.974073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974049 5943 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:03:39.974073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974067 5943 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:03:39.974195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974162 5943 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:03:39.974349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974338 5943 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:03:39.974460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974451 5943 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:03:39.974502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974490 5943 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:03:39.974531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974502 5943 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:03:39.974531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974512 5943 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:03:39.974531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974521 5943 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:03:39.974531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.974529 5943 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:03:39.975585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.975571 5943 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:03:39.975625 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.975588 5943 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:03:39.976271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.976172 5943 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:03:39.987527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987511 5943 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:03:39.987527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987525 5943 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:03:39.987668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987627 5943 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:03:39.987822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987810 5943 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:03:39.987852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987840 5943 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:03:39.987937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987928 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:03:39.987967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987942 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:03:39.987967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987949 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:03:39.987967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987955 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:03:39.987967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987961 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:03:39.987967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987967 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987973 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987979 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987986 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.987992 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.988000 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.988010 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.988034 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:03:39.988092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.988040 5943 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:03:39.991552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.991534 5943 server.go:1263] "Started kubelet" Oct 31 00:03:39.991792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.991639 5943 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:03:39.991792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.991764 5943 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:03:39.994476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.994118 5943 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:03:39.994656 ip-10-0-143-53 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:03:39.996286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.994709 5943 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:03:39.996876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:39.996420 5943 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:03:40.000117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000100 5943 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:03:40.000251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000113 5943 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:03:40.000251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000163 5943 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:28 +0000 UTC" deadline="2028-08-17 02:00:20.082247012 +0000 UTC" Oct 31 00:03:40.000251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000209 5943 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15745h56m40.082042449s" Oct 31 00:03:40.000251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000241 5943 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:03:40.000524 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000256 5943 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:03:40.000524 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.000342 5943 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:03:40.001230 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.000881 5943 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003365 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003414 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b5cca15-e32c-4fb2-8487-ace277115471" volumeName="kubernetes.io/projected/9b5cca15-e32c-4fb2-8487-ace277115471-kube-api-access-wwrl9" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003430 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003443 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce2530e-5ad4-4eec-ad2c-58bca81cae53" volumeName="kubernetes.io/configmap/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003456 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e406f6b-223a-4073-914b-74b5226b09c4" volumeName="kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-binary-copy" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003467 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003478 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003490 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003504 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dc42da-f207-4a5c-a481-6d5a65363802" volumeName="kubernetes.io/secret/f6dc42da-f207-4a5c-a481-6d5a65363802-metrics-tls" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003524 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003536 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0a53051-3a82-499c-bd29-9ccf93a535bc" volumeName="kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-tls" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003547 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003561 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc" volumeName="kubernetes.io/secret/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003574 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdac28b2-b646-4bcd-99d5-2c617094ea07" volumeName="kubernetes.io/projected/fdac28b2-b646-4bcd-99d5-2c617094ea07-kube-api-access-nswxp" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003588 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="499771b6-6b5b-4ad2-80e9-85f61d4d48e4" volumeName="kubernetes.io/configmap/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-serviceca" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003602 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0a53051-3a82-499c-bd29-9ccf93a535bc" volumeName="kubernetes.io/empty-dir/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003615 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61bcee6-5eb5-44b0-af5b-81bf788bf136" volumeName="kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003626 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc" volumeName="kubernetes.io/projected/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-kube-api-access-jb59b" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003638 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38f2d18c-7a3f-498f-b66e-b104cbab0e6c" volumeName="kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003650 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="955eeb9d-5b8c-473a-b0cf-69d6672353bf" volumeName="kubernetes.io/projected/955eeb9d-5b8c-473a-b0cf-69d6672353bf-kube-api-access-db6fr" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003662 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003694 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003706 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003719 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/projected/504be407-0ef5-4f9f-b6f5-5674004659b8-kube-api-access-jqzcf" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003731 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003744 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003756 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003769 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc" volumeName="kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-env-overrides" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003796 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="955eeb9d-5b8c-473a-b0cf-69d6672353bf" volumeName="kubernetes.io/secret/955eeb9d-5b8c-473a-b0cf-69d6672353bf-cert" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003808 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003820 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003832 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce2530e-5ad4-4eec-ad2c-58bca81cae53" volumeName="kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003843 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0a53051-3a82-499c-bd29-9ccf93a535bc" volumeName="kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003857 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003868 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dc42da-f207-4a5c-a481-6d5a65363802" volumeName="kubernetes.io/projected/f6dc42da-f207-4a5c-a481-6d5a65363802-kube-api-access-bqh9g" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003880 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bc7e49e-8890-47c1-a64f-25328161fde3" volumeName="kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003892 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003905 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="888e451d-b53f-487c-bd01-c90a6ad9bf8a" volumeName="kubernetes.io/projected/888e451d-b53f-487c-bd01-c90a6ad9bf8a-kube-api-access-z9rm4" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003917 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0a53051-3a82-499c-bd29-9ccf93a535bc" volumeName="kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003931 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e16f2f0-c342-414d-a8c8-5f68528c78b8" volumeName="kubernetes.io/empty-dir/5e16f2f0-c342-414d-a8c8-5f68528c78b8-data-volume" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003943 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003954 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003966 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e16f2f0-c342-414d-a8c8-5f68528c78b8" volumeName="kubernetes.io/secret/5e16f2f0-c342-414d-a8c8-5f68528c78b8-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003978 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.003990 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="499771b6-6b5b-4ad2-80e9-85f61d4d48e4" volumeName="kubernetes.io/projected/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-kube-api-access-5gzn2" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004002 5943 factory.go:55] Registering systemd factory Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004020 5943 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004003 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712ccf72-9a61-465b-a2df-495354882144" volumeName="kubernetes.io/empty-dir/712ccf72-9a61-465b-a2df-495354882144-tmp-dir" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004042 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61bcee6-5eb5-44b0-af5b-81bf788bf136" volumeName="kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004063 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004091 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cbf2b1a-032c-4047-81db-998713cc8927" volumeName="kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-tmp" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004104 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386772ab-964a-485d-9489-75e74e1fe1de" volumeName="kubernetes.io/configmap/386772ab-964a-485d-9489-75e74e1fe1de-konnectivity-ca" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004116 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cbf2b1a-032c-4047-81db-998713cc8927" volumeName="kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-etc-tuned" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004127 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386772ab-964a-485d-9489-75e74e1fe1de" volumeName="kubernetes.io/secret/386772ab-964a-485d-9489-75e74e1fe1de-agent-certs" seLinuxMountContext="" Oct 31 00:03:40.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004139 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004152 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004163 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004190 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce2530e-5ad4-4eec-ad2c-58bca81cae53" volumeName="kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004215 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004227 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004240 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004248 5943 factory.go:153] Registering CRI-O factory Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004252 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2" volumeName="kubernetes.io/projected/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-kube-api-access-pftp5" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004260 5943 factory.go:223] Registration of the crio container factory successfully Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004264 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="888e451d-b53f-487c-bd01-c90a6ad9bf8a" volumeName="kubernetes.io/secret/888e451d-b53f-487c-bd01-c90a6ad9bf8a-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004277 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b5cca15-e32c-4fb2-8487-ace277115471" volumeName="kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-cni-binary-copy" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004289 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61bcee6-5eb5-44b0-af5b-81bf788bf136" volumeName="kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004302 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e16f2f0-c342-414d-a8c8-5f68528c78b8" volumeName="kubernetes.io/projected/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-api-access-ljjm8" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004311 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-client-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004319 5943 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004340 5943 factory.go:103] Registering Raw factory Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004355 5943 manager.go:1199] Started watching for new ooms in manager Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004321 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004398 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004416 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004428 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dc42da-f207-4a5c-a481-6d5a65363802" volumeName="kubernetes.io/configmap/f6dc42da-f207-4a5c-a481-6d5a65363802-config-volume" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004439 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61bcee6-5eb5-44b0-af5b-81bf788bf136" volumeName="kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004451 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004460 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e406f6b-223a-4073-914b-74b5226b09c4" volumeName="kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004479 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61bcee6-5eb5-44b0-af5b-81bf788bf136" volumeName="kubernetes.io/projected/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-api-access-jztr5" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004487 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004494 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004503 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/configmap/2163faff-91f0-450a-aa4e-66336810549e-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004515 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004527 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e406f6b-223a-4073-914b-74b5226b09c4" volumeName="kubernetes.io/projected/9e406f6b-223a-4073-914b-74b5226b09c4-kube-api-access-mpvtj" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004544 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61bcee6-5eb5-44b0-af5b-81bf788bf136" volumeName="kubernetes.io/empty-dir/a61bcee6-5eb5-44b0-af5b-81bf788bf136-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004553 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004561 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0a53051-3a82-499c-bd29-9ccf93a535bc" volumeName="kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-metrics-client-ca" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004568 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004576 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e406f6b-223a-4073-914b-74b5226b09c4" volumeName="kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004584 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004593 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2" volumeName="kubernetes.io/configmap/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004604 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce2530e-5ad4-4eec-ad2c-58bca81cae53" volumeName="kubernetes.io/projected/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-kube-api-access-bg4pb" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004613 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004620 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004628 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/projected/822e1679-db28-423e-91c0-0dd4cf714902-kube-api-access-4qsk8" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004637 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-federate-client-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004645 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c04250d-a6cd-48e4-97b7-2959641907a7" volumeName="kubernetes.io/projected/7c04250d-a6cd-48e4-97b7-2959641907a7-kube-api-access-99cl7" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004653 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b5cca15-e32c-4fb2-8487-ace277115471" volumeName="kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-multus-daemon-config" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004661 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdac28b2-b646-4bcd-99d5-2c617094ea07" volumeName="kubernetes.io/empty-dir/fdac28b2-b646-4bcd-99d5-2c617094ea07-tmp" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004670 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e16f2f0-c342-414d-a8c8-5f68528c78b8" volumeName="kubernetes.io/configmap/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004686 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="725956a2-d668-4d42-989a-f5aeca0b2288" volumeName="kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004697 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822e1679-db28-423e-91c0-0dd4cf714902" volumeName="kubernetes.io/empty-dir/822e1679-db28-423e-91c0-0dd4cf714902-audit-log" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004705 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004712 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004721 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" volumeName="kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004728 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/projected/2163faff-91f0-450a-aa4e-66336810549e-kube-api-access-khcqh" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004737 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004745 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712ccf72-9a61-465b-a2df-495354882144" volumeName="kubernetes.io/projected/712ccf72-9a61-465b-a2df-495354882144-kube-api-access-mgsx7" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004753 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0a53051-3a82-499c-bd29-9ccf93a535bc" volumeName="kubernetes.io/projected/a0a53051-3a82-499c-bd29-9ccf93a535bc-kube-api-access-5hlbw" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004765 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004781 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5616844-000c-4b6c-8150-e7421eb3146c" volumeName="kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004789 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004804 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004812 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="725956a2-d668-4d42-989a-f5aeca0b2288" volumeName="kubernetes.io/projected/725956a2-d668-4d42-989a-f5aeca0b2288-kube-api-access-74p4g" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004819 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc" volumeName="kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-config" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004827 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dc42da-f207-4a5c-a481-6d5a65363802" volumeName="kubernetes.io/empty-dir/f6dc42da-f207-4a5c-a481-6d5a65363802-tmp-dir" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004838 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc" volumeName="kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004851 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cbf2b1a-032c-4047-81db-998713cc8927" volumeName="kubernetes.io/projected/1cbf2b1a-032c-4047-81db-998713cc8927-kube-api-access-x978l" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004875 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/empty-dir/2163faff-91f0-450a-aa4e-66336810549e-tmp" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004883 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2163faff-91f0-450a-aa4e-66336810549e" volumeName="kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004890 5943 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504be407-0ef5-4f9f-b6f5-5674004659b8" volumeName="kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-client-tls" seLinuxMountContext="" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004897 5943 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004902 5943 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:03:40.006541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.004630 5943 manager.go:319] Starting recovery of all containers Oct 31 00:03:40.031898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.031867 5943 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:03:40.033344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.033322 5943 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:03:40.033344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.033339 5943 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:03:40.033502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.033357 5943 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:03:40.033502 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.033390 5943 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:03:40.050645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.050615 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" exitCode=0 Oct 31 00:03:40.058345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.058311 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="af4412f13fe206bb249b210e92b982f1568dce2faa9567a05605ddce0c3ea805" exitCode=0 Oct 31 00:03:40.067549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.067517 5943 generic.go:356] "Generic (PLEG): container finished" podID="a0a53051-3a82-499c-bd29-9ccf93a535bc" containerID="641dd2dfce8cd4a12e2effa36ef0b71e3922b52a336fb4ff4ec1c6b3a442fcea" exitCode=0 Oct 31 00:03:40.078030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.078004 5943 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="8ec32c713d3c4392704480559c7c8fb55fafd3040cc822d063a54c255434f151" exitCode=0 Oct 31 00:03:40.078030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.078026 5943 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="a3621205af7df5e1ad1745648d6159baa263f3eca2adec5b9a8fa0cf25296c07" exitCode=0 Oct 31 00:03:40.078030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.078033 5943 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="66d76739c1b41b3fb6c107692cef862f9c3f32440a61c501e91fde2b45d0903d" exitCode=0 Oct 31 00:03:40.078241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.078038 5943 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="52c5d9d3e2fe18c3fd2520e65f2872a6f0b4414b5deb1c245f5bdc7326116011" exitCode=0 Oct 31 00:03:40.078241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.078043 5943 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="9655bc37dfadbdedcebba902e078c1c7e8e2628c288ec34116a28f025b976b67" exitCode=0 Oct 31 00:03:40.078241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.078047 5943 generic.go:356] "Generic (PLEG): container finished" podID="9e406f6b-223a-4073-914b-74b5226b09c4" containerID="396d4dbe3813bef756b908d46a280f6b42d682f1859866bc192c5ab8701de3c3" exitCode=0 Oct 31 00:03:40.089039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.088982 5943 generic.go:356] "Generic (PLEG): container finished" podID="87458182c04246bddfda72c933944155" containerID="e9c6631722e779d79dce94fc5da9424386a92e8cd9350d5e8745c50ac1b513fe" exitCode=0 Oct 31 00:03:40.100949 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.100926 5943 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" Oct 31 00:03:40.132496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.132475 5943 manager.go:324] Recovery completed Oct 31 00:03:40.133557 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.133537 5943 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:03:40.157516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.157496 5943 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:03:40.162981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.162961 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:03:40.163074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.162997 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:03:40.163074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.163011 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:03:40.166073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166049 5943 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:03:40.166073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166067 5943 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:03:40.166209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166076 5943 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:03:40.166209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166084 5943 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:03:40.166283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166239 5943 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:03:40.166283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166250 5943 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:03:40.166283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166272 5943 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:03:40.166283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166280 5943 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:03:40.166387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166289 5943 policy_none.go:50] "Start" Oct 31 00:03:40.166387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166305 5943 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:03:40.166387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166314 5943 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:03:40.166466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166450 5943 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:03:40.166466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166457 5943 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:03:40.166466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.166463 5943 policy_none.go:44] "Start" Oct 31 00:03:40.167464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.167448 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:03:40.167528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.167479 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:03:40.167528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.167493 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:03:40.171545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.171521 5943 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:03:40.174115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174101 5943 manager.go:341] "Starting Device Plugin manager" Oct 31 00:03:40.174191 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.174128 5943 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:03:40.174191 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174138 5943 server.go:87] "Starting device plugin registration server" Oct 31 00:03:40.174433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174421 5943 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:03:40.174463 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174437 5943 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:03:40.174557 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174541 5943 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:03:40.174686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174647 5943 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:03:40.174782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174698 5943 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:03:40.174782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.174705 5943 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:03:40.175699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.175609 5943 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:03:40.174680963Z","UUID":"8e323e29-128c-4a8e-99c4-f8eb066b7e63","Handler":null,"Name":"","Endpoint":""} Oct 31 00:03:40.176821 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.176486 5943 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:03:40.176821 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.176526 5943 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-53.us-west-1.compute.internal\" not found" Oct 31 00:03:40.177269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.177231 5943 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:03:40.177269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.177268 5943 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:03:40.186548 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.186530 5943 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:03:40.186880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.186865 5943 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:03:40.187201 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.187160 5943 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:03:40.235078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.235054 5943 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:03:40.275056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.275034 5943 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:03:40.276149 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.276130 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:03:40.276253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.276160 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:03:40.276253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.276170 5943 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:03:40.276326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.276260 5943 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.285079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.285062 5943 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:03:40.333830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.333765 5943 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal"] Oct 31 00:03:40.334544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334529 5943 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.334624 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334520 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"87458182c04246bddfda72c933944155","Type":"ContainerStarted","Data":"d3fdba430a26f86054d4f5accecae929c50992ed163f23287673318420f1ad48"} Oct 31 00:03:40.334676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334655 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"87458182c04246bddfda72c933944155","Type":"ContainerDied","Data":"e9c6631722e779d79dce94fc5da9424386a92e8cd9350d5e8745c50ac1b513fe"} Oct 31 00:03:40.334710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334686 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"87458182c04246bddfda72c933944155","Type":"ContainerStarted","Data":"41008fee89b4511399df098917e8b892c837e122cb2922c3c7d4f068da7af7e7"} Oct 31 00:03:40.334747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334707 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"8844070afaf498134d5a363201a59ac4","Type":"ContainerStarted","Data":"4b1c918fdcaf50e48d161c69a98ff98d0dc0d155be317c8fd2867933c13c4dbc"} Oct 31 00:03:40.334747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334737 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" event={"ID":"8844070afaf498134d5a363201a59ac4","Type":"ContainerStarted","Data":"89ef79be9cbbf2d70fbd4cd7f4e6ff53dc161c5e7b91911c5cb3c5d3524d7cc5"} Oct 31 00:03:40.334817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.334711 5943 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.386363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.386280 5943 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.386508 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.386371 5943 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.400903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.400879 5943 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:03:40.401044 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.400944 5943 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.401583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.401559 5943 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:03:40.401652 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:40.401590 5943 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.488872 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.488831 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.488872 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.488870 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8844070afaf498134d5a363201a59ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"8844070afaf498134d5a363201a59ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.489074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.488887 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.589358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.589321 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.589358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.589338 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.589551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.589390 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.589551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.589410 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8844070afaf498134d5a363201a59ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"8844070afaf498134d5a363201a59ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.589551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.589447 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8844070afaf498134d5a363201a59ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"8844070afaf498134d5a363201a59ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.589551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.589468 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87458182c04246bddfda72c933944155-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" (UID: \"87458182c04246bddfda72c933944155\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:40.974996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:40.974963 5943 apiserver.go:52] "Watching apiserver" Oct 31 00:03:41.080594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.080564 5943 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:03:41.081401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.081377 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs","openshift-cluster-node-tuning-operator/tuned-kt7ff","openshift-image-registry/node-ca-rlnr6","openshift-insights/insights-runtime-extractor-q4dds","openshift-dns/dns-default-p2dzg","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-69b47fd846-94zxw","openshift-multus/multus-vlv5b","kube-system/global-pull-secret-syncer-68spz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-4slbb","openshift-console/downloads-6dd789c8b7-nlmmj","openshift-monitoring/metrics-server-86d6c7656b-2hqmr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4","openshift-monitoring/monitoring-plugin-77f4f76594-st7jb","openshift-network-operator/iptables-alerter-6t9z7","openshift-ingress-canary/ingress-canary-nxhq2","openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg","openshift-multus/multus-additional-cni-plugins-vq95h","openshift-dns/node-resolver-5v8zm","openshift-ovn-kubernetes/ovnkube-node-8l9cg","openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr","kube-system/konnectivity-agent-4hztx","openshift-network-diagnostics/network-check-target-2kmrp","kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal","openshift-monitoring/node-exporter-67jm8"] Oct 31 00:03:41.103571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.103543 5943 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:03:41.112078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.112055 5943 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:41.112221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.112202 5943 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:41.135477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.135448 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt67z\"" Oct 31 00:03:41.137132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137100 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ddpr\"" Oct 31 00:03:41.137132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137109 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:03:41.137329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137254 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqqqs\"" Oct 31 00:03:41.137329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137282 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgjmv\"" Oct 31 00:03:41.137329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137304 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7xfz6\"" Oct 31 00:03:41.137329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137314 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\"" Oct 31 00:03:41.137488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137358 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsk29\"" Oct 31 00:03:41.137601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137516 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:03:41.137601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.137585 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27h5h\"" Oct 31 00:03:41.138890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.138872 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\"" Oct 31 00:03:41.139013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.138996 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gvgd9\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139039 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139041 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139069 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139076 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139106 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139130 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139137 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139143 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139151 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139195 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139201 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139204 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139217 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139221 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139296 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139482 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139497 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139485 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139539 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139539 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139550 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139565 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.139600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139582 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139619 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139626 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139643 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139652 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139668 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139696 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139815 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139864 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139911 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wr2gb\"" Oct 31 00:03:41.140148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.139977 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hplth\"" Oct 31 00:03:41.180379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180354 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.180379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180375 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:03:41.180632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180610 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.180632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180628 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.180775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180660 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:03:41.180861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180846 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:03:41.180903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.180846 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.181912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.181898 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 00:03:41.182858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.182844 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.183168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.183154 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.183237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.183193 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:03:41.183237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.183224 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:03:41.183308 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.183239 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:03:41.183308 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.183257 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:03:41.183387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.183306 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:03:41.186295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.186278 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:03:41.186455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.186442 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:03:41.186510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.186466 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:03:41.189484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.189467 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:03:41.189584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.189569 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:03:41.189723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.189712 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:03:41.189795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.189779 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:03:41.193243 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193227 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.193286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193251 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.193286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193271 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.193361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193287 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlbw\" (UniqueName: \"kubernetes.io/projected/a0a53051-3a82-499c-bd29-9ccf93a535bc-kube-api-access-5hlbw\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.193361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193308 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsx7\" (UniqueName: \"kubernetes.io/projected/712ccf72-9a61-465b-a2df-495354882144-kube-api-access-mgsx7\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.193361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193331 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.193361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193357 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-modprobe-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.193518 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193382 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a61bcee6-5eb5-44b0-af5b-81bf788bf136-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.193559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193531 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.193598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193565 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.193633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193600 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.193633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193616 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a61bcee6-5eb5-44b0-af5b-81bf788bf136-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.193736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193632 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.193736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193726 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-slash\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.193825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193744 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.193825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193759 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-sys\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.193825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193791 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-wtmp\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.193971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.193839 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dc42da-f207-4a5c-a481-6d5a65363802-tmp-dir\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.194328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194268 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdac28b2-b646-4bcd-99d5-2c617094ea07-tmp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:41.194409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194358 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdac28b2-b646-4bcd-99d5-2c617094ea07-tmp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:41.194409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194389 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6dc42da-f207-4a5c-a481-6d5a65363802-tmp-dir\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.194496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194390 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.194496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194463 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.194591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194499 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-netd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.194591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194527 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-root\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.194591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194554 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.194725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194643 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.194779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194752 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.194830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194793 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-hostroot\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.194878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194831 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-conf-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.194878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194865 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.194972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194903 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-lib-modules\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.194972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194796 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.194972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194925 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.194972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194943 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e16f2f0-c342-414d-a8c8-5f68528c78b8-crio-socket\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.195155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.194998 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.195155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195035 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955eeb9d-5b8c-473a-b0cf-69d6672353bf-cert\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:41.195155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195085 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-device-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.195326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195221 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-socket-dir-parent\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.195326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195262 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-multus-certs\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.195326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195307 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e16f2f0-c342-414d-a8c8-5f68528c78b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.195458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195444 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.195514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195454 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.195597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195570 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.195655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195643 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-systemd-units\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.195708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195676 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-log-socket\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.195764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195711 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955eeb9d-5b8c-473a-b0cf-69d6672353bf-cert\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:41.195764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195717 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/386772ab-964a-485d-9489-75e74e1fe1de-konnectivity-ca\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:03:41.195901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195882 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.195958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195925 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-kubelet\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.196007 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195970 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/386772ab-964a-485d-9489-75e74e1fe1de-konnectivity-ca\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:03:41.196007 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.195975 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rm4\" (UniqueName: \"kubernetes.io/projected/888e451d-b53f-487c-bd01-c90a6ad9bf8a-kube-api-access-z9rm4\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:41.196108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196008 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.196108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196037 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-cnibin\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.196108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196082 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4pb\" (UniqueName: \"kubernetes.io/projected/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-kube-api-access-bg4pb\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.196267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196143 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2163faff-91f0-450a-aa4e-66336810549e-tmp\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.196267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196202 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.196267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196239 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.196420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196311 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2163faff-91f0-450a-aa4e-66336810549e-tmp\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.196420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196357 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-etc-kubernetes\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.196420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196395 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.196566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196430 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p4g\" (UniqueName: \"kubernetes.io/projected/725956a2-d668-4d42-989a-f5aeca0b2288-kube-api-access-74p4g\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:41.196566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196456 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.196566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196481 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.196566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196490 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-conf\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.196566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196558 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-host\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.196795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196594 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.196795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196634 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-ovn\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.196795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196671 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-bin\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.196795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196706 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovn-node-metrics-cert\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.196795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196736 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-grpc-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.196795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196775 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712ccf72-9a61-465b-a2df-495354882144-hosts-file\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196817 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dc42da-f207-4a5c-a481-6d5a65363802-config-volume\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196856 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196892 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196930 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.196967 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197013 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-kubernetes\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.197069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197049 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-sys\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.197433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197085 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-host-slash\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.197433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197122 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrl9\" (UniqueName: \"kubernetes.io/projected/9b5cca15-e32c-4fb2-8487-ace277115471-kube-api-access-wwrl9\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.197433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197127 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dc42da-f207-4a5c-a481-6d5a65363802-config-volume\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.197433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197225 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.197433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197423 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/725956a2-d668-4d42-989a-f5aeca0b2288-metrics-certs\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:41.197666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197439 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovn-node-metrics-cert\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.197666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197494 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bc7e49e-8890-47c1-a64f-25328161fde3-original-pull-secret\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.197666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197552 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-metrics-client-ca\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.197666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197594 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-cnibin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.197666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197663 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-k8s-cni-cncf-io\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.197945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197698 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.197945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197749 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.197945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197844 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.197945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197848 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-systemd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.197945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197908 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712ccf72-9a61-465b-a2df-495354882144-tmp-dir\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.197945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197941 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-metrics-client-ca\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.198259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197945 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqh9g\" (UniqueName: \"kubernetes.io/projected/f6dc42da-f207-4a5c-a481-6d5a65363802-kube-api-access-bqh9g\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.198259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.197993 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.198259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198106 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.198259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198153 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712ccf72-9a61-465b-a2df-495354882144-tmp-dir\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.198259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198171 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.198259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198248 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysconfig\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.198519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198429 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.198519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198465 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.198519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198496 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.198658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198528 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/888e451d-b53f-487c-bd01-c90a6ad9bf8a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:41.198658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198558 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-serving-certs-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.198658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198588 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-metrics-client-ca\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.198658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198617 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.198658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198647 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.198869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198677 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-tmp\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.198869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198720 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-client-certs\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.198869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198752 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.198869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198786 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-var-lib-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.198869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198814 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.198869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198846 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzn2\" (UniqueName: \"kubernetes.io/projected/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-kube-api-access-5gzn2\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198877 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-var-lib-kubelet\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198909 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198938 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-env-overrides\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198964 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-script-lib\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.198994 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-systemd\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199025 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftp5\" (UniqueName: \"kubernetes.io/projected/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-kube-api-access-pftp5\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199057 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199082 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-run\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.199125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199113 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjm8\" (UniqueName: \"kubernetes.io/projected/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-api-access-ljjm8\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199145 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199194 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-host\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199228 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199255 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-netns\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199287 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199320 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-iptables-alerter-script\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199351 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199379 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-bin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.199595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199548 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-tmp\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.199996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199732 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.199996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199818 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-iptables-alerter-script\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.199996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199971 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-serving-certs-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.200107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.199994 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.200107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200038 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-federate-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.200107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200053 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-env-overrides\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.200107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200073 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/386772ab-964a-485d-9489-75e74e1fe1de-agent-certs\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:03:41.200107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200103 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200139 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-textfile\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200137 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61bcee6-5eb5-44b0-af5b-81bf788bf136-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200228 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200295 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e16f2f0-c342-414d-a8c8-5f68528c78b8-data-volume\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200301 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-textfile\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200323 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.200363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200352 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.200607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200388 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e16f2f0-c342-414d-a8c8-5f68528c78b8-data-volume\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.200607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200397 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.200607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200390 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-node-log\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.200607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200443 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e406f6b-223a-4073-914b-74b5226b09c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.200607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200583 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200608 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswxp\" (UniqueName: \"kubernetes.io/projected/fdac28b2-b646-4bcd-99d5-2c617094ea07-kube-api-access-nswxp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200583 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-script-lib\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200637 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-dbus\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200671 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200676 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/386772ab-964a-485d-9489-75e74e1fe1de-agent-certs\") pod \"konnectivity-agent-4hztx\" (UID: \"386772ab-964a-485d-9489-75e74e1fe1de\") " pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200699 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0a53051-3a82-499c-bd29-9ccf93a535bc-metrics-client-ca\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200777 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.200814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200773 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-sys-fs\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200854 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200887 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x978l\" (UniqueName: \"kubernetes.io/projected/1cbf2b1a-032c-4047-81db-998713cc8927-kube-api-access-x978l\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200911 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-multus-daemon-config\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200940 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6fr\" (UniqueName: \"kubernetes.io/projected/955eeb9d-5b8c-473a-b0cf-69d6672353bf-kube-api-access-db6fr\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.200969 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-client-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201004 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb59b\" (UniqueName: \"kubernetes.io/projected/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-kube-api-access-jb59b\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201037 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6dc42da-f207-4a5c-a481-6d5a65363802-metrics-tls\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.201097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201071 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-system-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201100 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-cni-binary-copy\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201117 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-multus-daemon-config\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201132 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201166 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/822e1679-db28-423e-91c0-0dd4cf714902-audit-log\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201226 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201257 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-tls\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201255 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/822e1679-db28-423e-91c0-0dd4cf714902-audit-log\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201283 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201299 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201351 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-etc-tuned\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201359 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5cca15-e32c-4fb2-8487-ace277115471-cni-binary-copy\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201394 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvtj\" (UniqueName: \"kubernetes.io/projected/9e406f6b-223a-4073-914b-74b5226b09c4-kube-api-access-mpvtj\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201414 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-tls\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201437 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-socket-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201436 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1cbf2b1a-032c-4047-81db-998713cc8927-etc-tuned\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201459 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201482 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.201541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201526 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201630 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201648 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201766 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201786 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201807 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201832 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-metrics-server-audit-profiles\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201866 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2163faff-91f0-450a-aa4e-66336810549e-metrics-client-ca\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201901 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.201979 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-multus\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202008 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztr5\" (UniqueName: \"kubernetes.io/projected/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-api-access-jztr5\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202036 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2163faff-91f0-450a-aa4e-66336810549e-metrics-client-ca\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202056 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsk8\" (UniqueName: \"kubernetes.io/projected/822e1679-db28-423e-91c0-0dd4cf714902-kube-api-access-4qsk8\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202070 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-metrics-server-audit-profiles\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202206 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202248 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzcf\" (UniqueName: \"kubernetes.io/projected/504be407-0ef5-4f9f-b6f5-5674004659b8-kube-api-access-jqzcf\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.202288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202287 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-config\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202323 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-kubelet\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202355 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-etc-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202385 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202428 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khcqh\" (UniqueName: \"kubernetes.io/projected/2163faff-91f0-450a-aa4e-66336810549e-kube-api-access-khcqh\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202452 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202478 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.202757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.202486 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-ovnkube-config\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.203140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203106 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.203565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203214 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-serviceca\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.203663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203590 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-registration-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.203663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203626 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cl7\" (UniqueName: \"kubernetes.io/projected/7c04250d-a6cd-48e4-97b7-2959641907a7-kube-api-access-99cl7\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.203663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203652 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.203663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203655 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.203853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203675 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-kubelet-config\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.203853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203706 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-netns\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.203853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203738 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.203853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203768 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822e1679-db28-423e-91c0-0dd4cf714902-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.203853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.203815 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.204047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.204030 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-serviceca\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.204224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.204208 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.232036 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.231961 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 00:03:41.232036 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.231974 5943 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:03:41.232036 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.232021 5943 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:03:41.232228 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:41.232051 5943 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:41.232228 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:41.232064 5943 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-53.us-west-1.compute.internal" Oct 31 00:03:41.232622 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.232604 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.237729 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.237706 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:03:41.237852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.237734 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:03:41.238197 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238158 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 00:03:41.238286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238209 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:03:41.238497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238453 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:03:41.238497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238476 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8ntdh\"" Oct 31 00:03:41.238497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238491 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:03:41.238644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238622 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:03:41.238644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238628 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:03:41.238644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238624 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238659 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238668 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238673 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238677 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238690 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238680 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238704 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:03:41.238775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238712 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:03:41.239047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238916 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:03:41.239047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.238971 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:03:41.239047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239008 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:03:41.239047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239014 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 00:03:41.239047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239039 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239049 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239059 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239103 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239107 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-adcr02j2qj3ro\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239112 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4gwp\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239127 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239116 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239143 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-89h7l\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239165 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239166 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rs266\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239233 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239240 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239255 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239273 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-27lg5\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239275 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wcptn\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239289 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r2q95\"" Oct 31 00:03:41.239295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239298 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239325 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239332 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5b339617vpk2d\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239340 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5lv6p\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239369 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239372 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239387 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-75f6pg955fqte\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239395 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vhzln\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239394 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sm2z2\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239418 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239424 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239421 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lb62x\"" Oct 31 00:03:41.239814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.239431 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtt85\"" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240671 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240694 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240699 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240746 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240767 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-federate-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240778 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-client-certs\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.240814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240806 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.241058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240806 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/888e451d-b53f-487c-bd01-c90a6ad9bf8a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:41.241058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240834 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.241058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.240860 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-secret-telemeter-client\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.242065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242038 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.242208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242048 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.242208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242132 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.242333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242218 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6dc42da-f207-4a5c-a481-6d5a65363802-metrics-tls\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.242333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242238 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.242333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242254 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-secret-metrics-server-tls\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.242333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242218 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.242333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242219 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/822e1679-db28-423e-91c0-0dd4cf714902-client-ca-bundle\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.242333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242278 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.242521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242323 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-tls\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.242521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.242373 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244592 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244615 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244665 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/504be407-0ef5-4f9f-b6f5-5674004659b8-telemeter-client-tls\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244671 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244663 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244672 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.244720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244666 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.244997 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244714 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.244997 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.244769 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.245902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.245887 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e16f2f0-c342-414d-a8c8-5f68528c78b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.246021 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.246010 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.246989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.246974 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.247108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.247090 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.247138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.247127 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2163faff-91f0-450a-aa4e-66336810549e-secret-grpc-tls\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.248259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.248241 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.248259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.248250 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.248349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.248257 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.248349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.248291 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.248411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.248375 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.253544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.253524 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsx7\" (UniqueName: \"kubernetes.io/projected/712ccf72-9a61-465b-a2df-495354882144-kube-api-access-mgsx7\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.253957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.253942 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlbw\" (UniqueName: \"kubernetes.io/projected/a0a53051-3a82-499c-bd29-9ccf93a535bc-kube-api-access-5hlbw\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.254621 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.254601 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rm4\" (UniqueName: \"kubernetes.io/projected/888e451d-b53f-487c-bd01-c90a6ad9bf8a-kube-api-access-z9rm4\") pod \"monitoring-plugin-77f4f76594-st7jb\" (UID: \"888e451d-b53f-487c-bd01-c90a6ad9bf8a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:41.254785 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.254767 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p4g\" (UniqueName: \"kubernetes.io/projected/725956a2-d668-4d42-989a-f5aeca0b2288-kube-api-access-74p4g\") pod \"network-metrics-daemon-4slbb\" (UID: \"725956a2-d668-4d42-989a-f5aeca0b2288\") " pod="openshift-multus/network-metrics-daemon-4slbb" Oct 31 00:03:41.254957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.254937 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4pb\" (UniqueName: \"kubernetes.io/projected/1ce2530e-5ad4-4eec-ad2c-58bca81cae53-kube-api-access-bg4pb\") pod \"openshift-state-metrics-7bd7489957-s8ddg\" (UID: \"1ce2530e-5ad4-4eec-ad2c-58bca81cae53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8ddg" Oct 31 00:03:41.256033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.256016 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrl9\" (UniqueName: \"kubernetes.io/projected/9b5cca15-e32c-4fb2-8487-ace277115471-kube-api-access-wwrl9\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.256254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.256239 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") pod \"prometheus-k8s-0\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:41.265526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.265499 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjm8\" (UniqueName: \"kubernetes.io/projected/5e16f2f0-c342-414d-a8c8-5f68528c78b8-kube-api-access-ljjm8\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.265603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.265575 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftp5\" (UniqueName: \"kubernetes.io/projected/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-kube-api-access-pftp5\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.265641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.265585 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqh9g\" (UniqueName: \"kubernetes.io/projected/f6dc42da-f207-4a5c-a481-6d5a65363802-kube-api-access-bqh9g\") pod \"dns-default-p2dzg\" (UID: \"f6dc42da-f207-4a5c-a481-6d5a65363802\") " pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:41.265686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.265637 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsk8\" (UniqueName: \"kubernetes.io/projected/822e1679-db28-423e-91c0-0dd4cf714902-kube-api-access-4qsk8\") pod \"metrics-server-86d6c7656b-2hqmr\" (UID: \"822e1679-db28-423e-91c0-0dd4cf714902\") " pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:03:41.265861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.265845 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwr7\" (UniqueName: \"kubernetes.io/projected/38f2d18c-7a3f-498f-b66e-b104cbab0e6c-kube-api-access-dkwr7\") pod \"network-check-target-2kmrp\" (UID: \"38f2d18c-7a3f-498f-b66e-b104cbab0e6c\") " pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:41.270896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.270872 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzn2\" (UniqueName: \"kubernetes.io/projected/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-kube-api-access-5gzn2\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.270998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.270869 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cl7\" (UniqueName: \"kubernetes.io/projected/7c04250d-a6cd-48e4-97b7-2959641907a7-kube-api-access-99cl7\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.270998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.270914 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6fr\" (UniqueName: \"kubernetes.io/projected/955eeb9d-5b8c-473a-b0cf-69d6672353bf-kube-api-access-db6fr\") pod \"ingress-canary-nxhq2\" (UID: \"955eeb9d-5b8c-473a-b0cf-69d6672353bf\") " pod="openshift-ingress-canary/ingress-canary-nxhq2" Oct 31 00:03:41.271149 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.271128 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") pod \"alertmanager-main-0\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:03:41.271242 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.271133 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswxp\" (UniqueName: \"kubernetes.io/projected/fdac28b2-b646-4bcd-99d5-2c617094ea07-kube-api-access-nswxp\") pod \"downloads-6dd789c8b7-nlmmj\" (UID: \"fdac28b2-b646-4bcd-99d5-2c617094ea07\") " pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:41.271242 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.271138 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztr5\" (UniqueName: \"kubernetes.io/projected/a61bcee6-5eb5-44b0-af5b-81bf788bf136-kube-api-access-jztr5\") pod \"kube-state-metrics-5dc8d7d89b-4txlr\" (UID: \"a61bcee6-5eb5-44b0-af5b-81bf788bf136\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4txlr" Oct 31 00:03:41.272479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.272456 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb59b\" (UniqueName: \"kubernetes.io/projected/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-kube-api-access-jb59b\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.272570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.272489 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvtj\" (UniqueName: \"kubernetes.io/projected/9e406f6b-223a-4073-914b-74b5226b09c4-kube-api-access-mpvtj\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.272570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.272493 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x978l\" (UniqueName: \"kubernetes.io/projected/1cbf2b1a-032c-4047-81db-998713cc8927-kube-api-access-x978l\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.272825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.272808 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzcf\" (UniqueName: \"kubernetes.io/projected/504be407-0ef5-4f9f-b6f5-5674004659b8-kube-api-access-jqzcf\") pod \"telemeter-client-7c8d84c7d8-5v8zs\" (UID: \"504be407-0ef5-4f9f-b6f5-5674004659b8\") " pod="openshift-monitoring/telemeter-client-7c8d84c7d8-5v8zs" Oct 31 00:03:41.272874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.272808 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khcqh\" (UniqueName: \"kubernetes.io/projected/2163faff-91f0-450a-aa4e-66336810549e-kube-api-access-khcqh\") pod \"thanos-querier-69b47fd846-94zxw\" (UID: \"2163faff-91f0-450a-aa4e-66336810549e\") " pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:41.304543 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304512 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-host\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.304636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304548 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-netns\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.304636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304589 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-bin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.304636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304620 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304639 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304670 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-node-log\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304687 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-dbus\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304710 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304688 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/499771b6-6b5b-4ad2-80e9-85f61d4d48e4-host\") pod \"node-ca-rlnr6\" (UID: \"499771b6-6b5b-4ad2-80e9-85f61d4d48e4\") " pod="openshift-image-registry/node-ca-rlnr6" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304729 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304739 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-node-log\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304737 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-sys-fs\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304708 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304721 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-netns\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304721 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-bin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304766 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-dbus\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304768 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.304797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304784 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-sys-fs\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304870 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-system-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304908 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-system-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304939 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.304982 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-socket-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305017 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-multus\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305024 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305045 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-kubelet\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305064 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-socket-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305072 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-etc-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305103 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-kubelet\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305106 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-cni-multus\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305103 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-registration-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305145 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-kubelet-config\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305162 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-netns\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305143 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-etc-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305209 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-registration-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305209 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-modprobe-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305244 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-netns\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305252 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305222 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bc7e49e-8890-47c1-a64f-25328161fde3-kubelet-config\") pod \"global-pull-secret-syncer-68spz\" (UID: \"1bc7e49e-8890-47c1-a64f-25328161fde3\") " pod="kube-system/global-pull-secret-syncer-68spz" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305269 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-slash\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305297 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-slash\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305298 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305319 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-sys\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305331 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305317 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-modprobe-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305336 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-wtmp\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305359 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-d\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305378 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305384 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-sys\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305384 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-node-exporter-wtmp\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305401 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-netd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305420 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-root\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.305421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305437 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-hostroot\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305452 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-conf-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305464 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-netd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305469 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-lib-modules\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305479 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0a53051-3a82-499c-bd29-9ccf93a535bc-root\") pod \"node-exporter-67jm8\" (UID: \"a0a53051-3a82-499c-bd29-9ccf93a535bc\") " pod="openshift-monitoring/node-exporter-67jm8" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305491 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-hostroot\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305500 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-conf-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305506 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e16f2f0-c342-414d-a8c8-5f68528c78b8-crio-socket\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305522 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-cni-dir\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305538 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-device-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305542 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e16f2f0-c342-414d-a8c8-5f68528c78b8-crio-socket\") pod \"insights-runtime-extractor-q4dds\" (UID: \"5e16f2f0-c342-414d-a8c8-5f68528c78b8\") " pod="openshift-insights/insights-runtime-extractor-q4dds" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305560 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-socket-dir-parent\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305564 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-lib-modules\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305589 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-multus-certs\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305598 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c04250d-a6cd-48e4-97b7-2959641907a7-device-dir\") pod \"aws-ebs-csi-driver-node-bmsp4\" (UID: \"7c04250d-a6cd-48e4-97b7-2959641907a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmsp4" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305609 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-multus-socket-dir-parent\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305605 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-systemd-units\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305625 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-systemd-units\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305627 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-multus-certs\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305640 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-log-socket\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305659 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-kubelet\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305673 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-cnibin\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305694 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-log-socket\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305713 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-var-lib-kubelet\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305716 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-cnibin\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305744 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-etc-kubernetes\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305760 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-conf\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305788 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-etc-kubernetes\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305788 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-host\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305810 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-ovn\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305815 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-host\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305825 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-bin\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305843 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712ccf72-9a61-465b-a2df-495354882144-hosts-file\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305848 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-ovn\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305851 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysctl-conf\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305863 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-kubernetes\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305882 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-cni-bin\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305887 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-sys\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305900 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-kubernetes\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305909 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-host-slash\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305899 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712ccf72-9a61-465b-a2df-495354882144-hosts-file\") pod \"node-resolver-5v8zm\" (UID: \"712ccf72-9a61-465b-a2df-495354882144\") " pod="openshift-dns/node-resolver-5v8zm" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305933 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-sys\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305933 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305958 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2-host-slash\") pod \"iptables-alerter-6t9z7\" (UID: \"106eb17b-35db-4dc6-9eb7-9c3c1b58a0d2\") " pod="openshift-network-operator/iptables-alerter-6t9z7" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305967 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-cnibin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305967 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e406f6b-223a-4073-914b-74b5226b09c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vq95h\" (UID: \"9e406f6b-223a-4073-914b-74b5226b09c4\") " pod="openshift-multus/multus-additional-cni-plugins-vq95h" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.305993 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-k8s-cni-cncf-io\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306019 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-systemd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306024 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-cnibin\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306041 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysconfig\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306071 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-var-lib-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306085 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b5cca15-e32c-4fb2-8487-ace277115471-host-run-k8s-cni-cncf-io\") pod \"multus-vlv5b\" (UID: \"9b5cca15-e32c-4fb2-8487-ace277115471\") " pod="openshift-multus/multus-vlv5b" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306095 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306110 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-var-lib-openvswitch\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306120 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-var-lib-kubelet\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306125 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306111 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-sysconfig\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306087 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc-run-systemd\") pod \"ovnkube-node-8l9cg\" (UID: \"f21baed8-0ec6-4331-b9a1-cc6f14dfa3fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306150 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-systemd\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306158 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-var-lib-kubelet\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306171 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-run\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306238 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-etc-systemd\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.306593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.306242 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1cbf2b1a-032c-4047-81db-998713cc8927-run\") pod \"tuned-kt7ff\" (UID: \"1cbf2b1a-032c-4047-81db-998713cc8927\") " pod="openshift-cluster-node-tuning-operator/tuned-kt7ff" Oct 31 00:03:41.399163 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.399143 5943 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:03:41.647831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.647794 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:41.793311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:41.793266 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:42.089306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:42.089267 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:42.205099 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:42.205065 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:03:42.251165 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:42.251131 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:43.150436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:43.150401 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:43.199271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:43.199236 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p2dzg" Oct 31 00:03:43.295245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:43.295210 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:44.206375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:44.206339 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:44.770699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:44.770664 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:44.789711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:44.789684 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:03:45.230593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:45.230503 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:45.424908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:45.424853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:46.285138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:46.285108 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69b47fd846-94zxw" Oct 31 00:03:48.450780 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:48.450749 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7jb" Oct 31 00:03:48.712595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:48.712523 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:03:48.712789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:48.712776 5943 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:03:50.548309 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:50.548269 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:50.738503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:50.738465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:51.504434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:51.504398 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hztx" Oct 31 00:03:53.551171 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:53.551132 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:53.691981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:53.691874 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:56.410433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:56.410383 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:56.557641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:56.557606 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:57.474619 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:03:57.474585 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:03:58.189439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:58.189391 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nlmmj" event={"ID":"fdac28b2-b646-4bcd-99d5-2c617094ea07","Type":"ContainerStarted","Data":"49f4500a97b4e11f88c2df965fa2a44b5e3536067eb06061cb3798f1c2b2e36b"} Oct 31 00:03:58.189747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:58.189718 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:58.208748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:58.208713 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nlmmj" Oct 31 00:03:59.153237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:59.153169 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:59.299815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:03:59.299780 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:00.970455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:00.970418 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86d6c7656b-2hqmr" Oct 31 00:04:01.559144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:01.559112 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:01.703054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:01.703014 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:02.891143 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:02.891107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:03.057201 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:03.057147 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:07.810045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:07.810020 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8l9cg" Oct 31 00:04:07.995360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:07.995328 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:08.138852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:08.138749 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:08.611938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:08.611910 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:09.733342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:09.733302 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:09.880904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:09.880872 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:10.030258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.030211 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:10.527566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.527530 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:04:10.528079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.528049 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="alertmanager" containerID="cri-o://832ed388a17b39340d188b8181d4452941009d71554f1f583d1098db63de592e" gracePeriod=120 Oct 31 00:04:10.528145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.528078 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-web" containerID="cri-o://142322dff666eeb4d1d06367790c8a7f3ef69540504cb6fd1730ec8751cf635b" gracePeriod=120 Oct 31 00:04:10.528145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.528102 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy" containerID="cri-o://a48d17a0f476314c5e5b588c64530e0499722487853dd1315961d904902baa4a" gracePeriod=120 Oct 31 00:04:10.528268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.528116 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="prom-label-proxy" containerID="cri-o://b3ee241e93217d77933eaa1b1b619d8e1030d9861889ab89eb85dfa5fc3cd38e" gracePeriod=120 Oct 31 00:04:10.528268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.528145 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="config-reloader" containerID="cri-o://d4f569c79edac59f760501c8bae071807aed5d82eed6e341d7b94de0e87d1649" gracePeriod=120 Oct 31 00:04:10.528268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:10.528060 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-metric" containerID="cri-o://a8358481b86a5b6e9d8debe9595ff71b9194544626b76205205f26503bb15a5c" gracePeriod=120 Oct 31 00:04:11.244779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244747 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="b3ee241e93217d77933eaa1b1b619d8e1030d9861889ab89eb85dfa5fc3cd38e" exitCode=0 Oct 31 00:04:11.244779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244770 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="a8358481b86a5b6e9d8debe9595ff71b9194544626b76205205f26503bb15a5c" exitCode=0 Oct 31 00:04:11.244779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244776 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="a48d17a0f476314c5e5b588c64530e0499722487853dd1315961d904902baa4a" exitCode=0 Oct 31 00:04:11.244779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244782 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="d4f569c79edac59f760501c8bae071807aed5d82eed6e341d7b94de0e87d1649" exitCode=0 Oct 31 00:04:11.245264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244788 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="832ed388a17b39340d188b8181d4452941009d71554f1f583d1098db63de592e" exitCode=0 Oct 31 00:04:11.245264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244816 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"b3ee241e93217d77933eaa1b1b619d8e1030d9861889ab89eb85dfa5fc3cd38e"} Oct 31 00:04:11.245264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244880 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"a8358481b86a5b6e9d8debe9595ff71b9194544626b76205205f26503bb15a5c"} Oct 31 00:04:11.245264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244905 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"a48d17a0f476314c5e5b588c64530e0499722487853dd1315961d904902baa4a"} Oct 31 00:04:11.245264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244928 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"d4f569c79edac59f760501c8bae071807aed5d82eed6e341d7b94de0e87d1649"} Oct 31 00:04:11.245264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:11.244952 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"832ed388a17b39340d188b8181d4452941009d71554f1f583d1098db63de592e"} Oct 31 00:04:12.208509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.208484 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2kmrp" Oct 31 00:04:12.252607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.252566 5943 generic.go:356] "Generic (PLEG): container finished" podID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerID="142322dff666eeb4d1d06367790c8a7f3ef69540504cb6fd1730ec8751cf635b" exitCode=0 Oct 31 00:04:12.253005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.252636 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"142322dff666eeb4d1d06367790c8a7f3ef69540504cb6fd1730ec8751cf635b"} Oct 31 00:04:12.310510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.310469 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:12.407920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.407835 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.407920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.407878 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.407920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.407914 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.407953 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.407983 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408024 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408068 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408098 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408124 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408206 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408269 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408299 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408345 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") pod \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\" (UID: \"e0c6d3e4-f576-4666-815d-f3aea0c5dab1\") " Oct 31 00:04:12.408603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408347 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:12.408603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.408411 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:12.409266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.409078 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:12.409266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.409197 5943 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.409266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.409220 5943 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-metrics-client-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.409266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.409237 5943 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-alertmanager-main-db\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.411244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.411217 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.411552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.411528 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.412001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.411977 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.412262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.412236 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:12.412401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.412375 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "kube-api-access-zv6bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:12.412818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.412792 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:12.412966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.412940 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.413430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.413402 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.416496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.416413 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.424419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.424387 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config" pod "e0c6d3e4-f576-4666-815d-f3aea0c5dab1" (UID: "e0c6d3e4-f576-4666-815d-f3aea0c5dab1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:12.510072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510040 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6bg\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-kube-api-access-zv6bg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510068 5943 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510079 5943 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-cluster-tls-config\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510088 5943 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510097 5943 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-tls-assets\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510107 5943 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-main-tls\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510116 5943 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510126 5943 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-config-out\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510137 5943 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-web-config\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.510322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:12.510155 5943 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0c6d3e4-f576-4666-815d-f3aea0c5dab1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:13.259779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.259748 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0c6d3e4-f576-4666-815d-f3aea0c5dab1","Type":"ContainerDied","Data":"f0ea0df8a8e3849c28c1e7dc9513869cb793bd5d61f267d12291417f5ad92430"} Oct 31 00:04:13.260344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.259799 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.260344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.259865 5943 scope.go:122] "RemoveContainer" containerID="b3ee241e93217d77933eaa1b1b619d8e1030d9861889ab89eb85dfa5fc3cd38e" Oct 31 00:04:13.268296 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.268277 5943 scope.go:122] "RemoveContainer" containerID="a8358481b86a5b6e9d8debe9595ff71b9194544626b76205205f26503bb15a5c" Oct 31 00:04:13.275551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.275535 5943 scope.go:122] "RemoveContainer" containerID="a48d17a0f476314c5e5b588c64530e0499722487853dd1315961d904902baa4a" Oct 31 00:04:13.282395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.282377 5943 scope.go:122] "RemoveContainer" containerID="142322dff666eeb4d1d06367790c8a7f3ef69540504cb6fd1730ec8751cf635b" Oct 31 00:04:13.289376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.289358 5943 scope.go:122] "RemoveContainer" containerID="d4f569c79edac59f760501c8bae071807aed5d82eed6e341d7b94de0e87d1649" Oct 31 00:04:13.296763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.296746 5943 scope.go:122] "RemoveContainer" containerID="832ed388a17b39340d188b8181d4452941009d71554f1f583d1098db63de592e" Oct 31 00:04:13.304231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.304210 5943 scope.go:122] "RemoveContainer" containerID="af4412f13fe206bb249b210e92b982f1568dce2faa9567a05605ddce0c3ea805" Oct 31 00:04:13.474234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.474199 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:04:13.523933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.523838 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:04:13.524285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524266 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:04:13.524557 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524545 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="alertmanager" Oct 31 00:04:13.524592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524558 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="alertmanager" Oct 31 00:04:13.524592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524571 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="prom-label-proxy" Oct 31 00:04:13.524592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524575 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="prom-label-proxy" Oct 31 00:04:13.524592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524588 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="init-config-reloader" Oct 31 00:04:13.524592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524593 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="init-config-reloader" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524603 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-web" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524607 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-web" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524615 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-metric" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524619 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-metric" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524626 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524630 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524644 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="config-reloader" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524648 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="config-reloader" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524693 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-web" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524701 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy-metric" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524707 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="kube-rbac-proxy" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524714 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="config-reloader" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524721 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="alertmanager" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524726 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="prom-label-proxy" Oct 31 00:04:13.524736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.524733 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" containerName="init-config-reloader" Oct 31 00:04:13.550911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.550879 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.603868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.603843 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:04:13.604068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.603931 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:04:13.604068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.603966 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:04:13.604068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.604052 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:04:13.604222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.604128 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:04:13.604372 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.604356 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:04:13.604458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.604439 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:04:13.604458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.604451 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:04:13.604544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.604479 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-27lg5\"" Oct 31 00:04:13.704736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.704706 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:04:13.720149 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720123 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e93e59-7eef-46cc-8acc-9d5bb3871736-config-out\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720164 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-web-config\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720271 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e93e59-7eef-46cc-8acc-9d5bb3871736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720305 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-config-volume\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720323 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720339 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63e93e59-7eef-46cc-8acc-9d5bb3871736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720433 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720468 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720493 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720513 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e93e59-7eef-46cc-8acc-9d5bb3871736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720651 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720696 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e93e59-7eef-46cc-8acc-9d5bb3871736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.720782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.720763 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzp9l\" (UniqueName: \"kubernetes.io/projected/63e93e59-7eef-46cc-8acc-9d5bb3871736-kube-api-access-lzp9l\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.769723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.769686 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:04:13.821583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821503 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-web-config\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821535 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e93e59-7eef-46cc-8acc-9d5bb3871736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821555 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-config-volume\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821571 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821590 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63e93e59-7eef-46cc-8acc-9d5bb3871736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821622 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821655 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821685 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821714 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e93e59-7eef-46cc-8acc-9d5bb3871736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821746 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821775 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e93e59-7eef-46cc-8acc-9d5bb3871736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821807 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzp9l\" (UniqueName: \"kubernetes.io/projected/63e93e59-7eef-46cc-8acc-9d5bb3871736-kube-api-access-lzp9l\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.821885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821871 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e93e59-7eef-46cc-8acc-9d5bb3871736-config-out\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.822343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.821966 5943 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:04:13.822343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.822091 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63e93e59-7eef-46cc-8acc-9d5bb3871736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.822950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.822832 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e93e59-7eef-46cc-8acc-9d5bb3871736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.822950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.822881 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e93e59-7eef-46cc-8acc-9d5bb3871736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.825192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.825141 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e93e59-7eef-46cc-8acc-9d5bb3871736-config-out\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.825597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.825526 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-web-config\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.825750 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.825731 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.825853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.825780 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e93e59-7eef-46cc-8acc-9d5bb3871736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.825853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.825831 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.826169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.826146 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-config-volume\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.826274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.826230 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.826274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.826236 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.827018 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.827001 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63e93e59-7eef-46cc-8acc-9d5bb3871736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.884144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.884113 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzp9l\" (UniqueName: \"kubernetes.io/projected/63e93e59-7eef-46cc-8acc-9d5bb3871736-kube-api-access-lzp9l\") pod \"alertmanager-main-0\" (UID: \"63e93e59-7eef-46cc-8acc-9d5bb3871736\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:13.965861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:13.965817 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:04:14.040526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:14.040495 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c6d3e4-f576-4666-815d-f3aea0c5dab1" path="/var/lib/kubelet/pods/e0c6d3e4-f576-4666-815d-f3aea0c5dab1/volumes" Oct 31 00:04:14.466596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:14.466563 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:04:14.470476 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:04:14.470448 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e93e59_7eef_46cc_8acc_9d5bb3871736.slice/crio-90b62bb925f68e131c577cc42df999c3d84f4a0a4dc7310454e8e7b4817c5011 WatchSource:0}: Error finding container 90b62bb925f68e131c577cc42df999c3d84f4a0a4dc7310454e8e7b4817c5011: Status 404 returned error can't find the container with id 90b62bb925f68e131c577cc42df999c3d84f4a0a4dc7310454e8e7b4817c5011 Oct 31 00:04:15.269046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:15.269011 5943 generic.go:356] "Generic (PLEG): container finished" podID="63e93e59-7eef-46cc-8acc-9d5bb3871736" containerID="aefb3596064c37d24114d2b88df09e096c7be61a1c51c1c698d399ffb2f9bb24" exitCode=0 Oct 31 00:04:15.269046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:15.269045 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerDied","Data":"aefb3596064c37d24114d2b88df09e096c7be61a1c51c1c698d399ffb2f9bb24"} Oct 31 00:04:15.269440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:15.269100 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"90b62bb925f68e131c577cc42df999c3d84f4a0a4dc7310454e8e7b4817c5011"} Oct 31 00:04:16.276386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.276354 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"3dab97395f2b052adda43a9486a632910e75e77742d9df22c11c3eee544f5f8f"} Oct 31 00:04:16.276901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.276417 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"60e5e3365c39b27aa2a7bef900933c479ed048690c69cbc3fdfdd71e72da6d0b"} Oct 31 00:04:16.276901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.276444 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"c4c739698dcedd4897a7dc0a36f15e2e4cc4eb828d7ed4f13fbd3b0775dfb01f"} Oct 31 00:04:16.276901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.276469 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"db72645c308077a054265a51a4d6ef7adf5c7feaefe9a8a496ccaa50868fbba4"} Oct 31 00:04:16.276901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.276498 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"e466d8000756b0ca1ed3b6067bf963c31884f2ec1df5ecae3bc9796af2e6cacd"} Oct 31 00:04:16.276901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.276526 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63e93e59-7eef-46cc-8acc-9d5bb3871736","Type":"ContainerStarted","Data":"81cc18ac2b76ffedcfec4eb2d5cc2879bfa49f09732aee7cef24b22bbb18761a"} Oct 31 00:04:16.279244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279215 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:04:16.279514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279480 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="prometheus" containerID="cri-o://b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" gracePeriod=600 Oct 31 00:04:16.279514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279504 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="thanos-sidecar" containerID="cri-o://f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" gracePeriod=600 Oct 31 00:04:16.279695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279504 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy" containerID="cri-o://df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" gracePeriod=600 Oct 31 00:04:16.279695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279518 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-web" containerID="cri-o://91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" gracePeriod=600 Oct 31 00:04:16.279695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279501 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" gracePeriod=600 Oct 31 00:04:16.279695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.279553 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="config-reloader" containerID="cri-o://b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" gracePeriod=600 Oct 31 00:04:16.407304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.407244 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.392901571 podStartE2EDuration="3.407228682s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.327111ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:16.406095845 +0000 UTC m=+36.499386066" watchObservedRunningTime="2026-10-31 00:04:16.407228682 +0000 UTC m=+36.500518863" Oct 31 00:04:16.583572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.583542 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:16.749612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749578 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.749824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749637 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.749824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749665 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.749824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749689 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.749824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749719 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.749824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749755 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749838 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749934 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.749973 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750005 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750033 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750081 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750117 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750158 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750203 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750221 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750259 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750287 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750317 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.750523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750391 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.751222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.750593 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") pod \"b5616844-000c-4b6c-8150-e7421eb3146c\" (UID: \"b5616844-000c-4b6c-8150-e7421eb3146c\") " Oct 31 00:04:16.751443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.751410 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.751533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.751477 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.751921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.751893 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.752409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752126 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752677 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752844 5943 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752866 5943 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752883 5943 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-metrics-client-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752900 5943 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752916 5943 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752923 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:16.752960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.752931 5943 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.753455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.753131 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.754355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.754328 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:16.754474 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.754447 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:16.754551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.754518 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "kube-api-access-jwbqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:16.754612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.754529 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.754716 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.754603 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.754911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.754874 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.755139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.755113 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.755582 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.755550 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.755845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.755824 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.756357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.756339 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.756747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.756733 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:16.765653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.765631 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config" pod "b5616844-000c-4b6c-8150-e7421eb3146c" (UID: "b5616844-000c-4b6c-8150-e7421eb3146c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:16.853773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853738 5943 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853771 5943 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853782 5943 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853792 5943 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-config-out\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853801 5943 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5616844-000c-4b6c-8150-e7421eb3146c-prometheus-k8s-db\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853810 5943 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853819 5943 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853836 5943 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-config\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853846 5943 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-web-config\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853854 5943 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-kube-rbac-proxy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853863 5943 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-tls-assets\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853871 5943 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-grpc-tls\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853879 5943 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5616844-000c-4b6c-8150-e7421eb3146c-secret-metrics-client-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:16.853967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:16.853887 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbqw\" (UniqueName: \"kubernetes.io/projected/b5616844-000c-4b6c-8150-e7421eb3146c-kube-api-access-jwbqw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:17.283585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283544 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" exitCode=0 Oct 31 00:04:17.283585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283574 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" exitCode=0 Oct 31 00:04:17.283585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283584 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" exitCode=0 Oct 31 00:04:17.283585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283592 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" exitCode=0 Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283600 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" exitCode=0 Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283594 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283640 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283668 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283607 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5616844-000c-4b6c-8150-e7421eb3146c" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" exitCode=0 Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283696 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283723 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283755 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283781 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283805 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5616844-000c-4b6c-8150-e7421eb3146c","Type":"ContainerDied","Data":"d9aa1edab3625a35d423dba9b4979bfbe15b2388b5bbeeb55a8376ab57af75fb"} Oct 31 00:04:17.284259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.283754 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.292468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.292451 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.299464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.299441 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.306127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.306106 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.312945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.312920 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.319797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.319782 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.332953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.332936 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.340278 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.340262 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.340546 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.340523 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.340599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.340553 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} err="failed to get container status \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" Oct 31 00:04:17.340599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.340593 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.340852 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.340836 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.340906 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.340859 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} err="failed to get container status \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" Oct 31 00:04:17.340906 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.340875 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.341094 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.341079 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.341135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341107 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} err="failed to get container status \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" Oct 31 00:04:17.341135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341120 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.341425 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.341408 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.341499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341430 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} err="failed to get container status \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" Oct 31 00:04:17.341499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341449 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.341687 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.341671 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.341749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341700 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} err="failed to get container status \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" Oct 31 00:04:17.341749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341713 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.341974 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.341958 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.342015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341980 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} err="failed to get container status \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" Oct 31 00:04:17.342015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.341994 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.342509 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:17.342491 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.342553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.342514 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} err="failed to get container status \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" Oct 31 00:04:17.342553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.342531 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.342751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.342735 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} err="failed to get container status \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" Oct 31 00:04:17.342791 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.342752 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.342972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.342952 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} err="failed to get container status \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" Oct 31 00:04:17.343012 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.342972 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.343211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343195 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} err="failed to get container status \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" Oct 31 00:04:17.343211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343211 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.343420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343404 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} err="failed to get container status \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" Oct 31 00:04:17.343465 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343420 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.343666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343649 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} err="failed to get container status \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" Oct 31 00:04:17.343704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343666 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.343869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343853 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} err="failed to get container status \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" Oct 31 00:04:17.343906 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.343869 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.344098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344081 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} err="failed to get container status \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" Oct 31 00:04:17.344098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344097 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.344343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344325 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} err="failed to get container status \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" Oct 31 00:04:17.344343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344343 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.344556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344536 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} err="failed to get container status \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" Oct 31 00:04:17.344604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344557 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.344782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344763 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} err="failed to get container status \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" Oct 31 00:04:17.344836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344781 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.344983 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344967 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} err="failed to get container status \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" Oct 31 00:04:17.345031 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.344983 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.345210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345194 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} err="failed to get container status \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" Oct 31 00:04:17.345250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345210 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.345411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345395 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} err="failed to get container status \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" Oct 31 00:04:17.345452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345412 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.345628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345613 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} err="failed to get container status \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" Oct 31 00:04:17.345628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345628 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.345852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345835 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} err="failed to get container status \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" Oct 31 00:04:17.345852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.345851 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.346062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346045 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} err="failed to get container status \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" Oct 31 00:04:17.346062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346061 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.346314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346296 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} err="failed to get container status \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" Oct 31 00:04:17.346362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346315 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.346541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346523 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} err="failed to get container status \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" Oct 31 00:04:17.346603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346541 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.346758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346742 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} err="failed to get container status \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" Oct 31 00:04:17.346758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346758 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.346964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346945 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} err="failed to get container status \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" Oct 31 00:04:17.346964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.346964 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.347194 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347165 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} err="failed to get container status \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" Oct 31 00:04:17.347256 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347200 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.347420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347405 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} err="failed to get container status \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" Oct 31 00:04:17.347420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347419 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.347631 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347615 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} err="failed to get container status \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" Oct 31 00:04:17.347677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347631 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.347878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347863 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} err="failed to get container status \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" Oct 31 00:04:17.347923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.347877 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.348070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348051 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} err="failed to get container status \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" Oct 31 00:04:17.348107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348070 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.348307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348291 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} err="failed to get container status \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" Oct 31 00:04:17.348307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348307 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.348538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348521 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} err="failed to get container status \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" Oct 31 00:04:17.348584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348538 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.348745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348730 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} err="failed to get container status \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" Oct 31 00:04:17.348784 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348745 5943 scope.go:122] "RemoveContainer" containerID="9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf" Oct 31 00:04:17.348963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348946 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf"} err="failed to get container status \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": rpc error: code = NotFound desc = could not find container \"9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf\": container with ID starting with 9c3269304c387771d1fbbcbbc64722fb1a3165ee402c92336a48bca2dbc347cf not found: ID does not exist" Oct 31 00:04:17.348963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.348963 5943 scope.go:122] "RemoveContainer" containerID="df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75" Oct 31 00:04:17.349196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349165 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75"} err="failed to get container status \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": rpc error: code = NotFound desc = could not find container \"df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75\": container with ID starting with df8d6cf66a9d4639ac4fc93b6416d80eae5e833d40669353cb8a099b51a7fc75 not found: ID does not exist" Oct 31 00:04:17.349196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349192 5943 scope.go:122] "RemoveContainer" containerID="91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e" Oct 31 00:04:17.349392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349378 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e"} err="failed to get container status \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": rpc error: code = NotFound desc = could not find container \"91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e\": container with ID starting with 91bba67725b033356683a205211a41550ca99ad9b84687fc63e4d21a4e2ab69e not found: ID does not exist" Oct 31 00:04:17.349392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349391 5943 scope.go:122] "RemoveContainer" containerID="f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756" Oct 31 00:04:17.349571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349552 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756"} err="failed to get container status \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": rpc error: code = NotFound desc = could not find container \"f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756\": container with ID starting with f1b8a74ebd9336c4a2b81e165ac8f91b4c6634031c67c4b2738a7ed683344756 not found: ID does not exist" Oct 31 00:04:17.349571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349567 5943 scope.go:122] "RemoveContainer" containerID="b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768" Oct 31 00:04:17.349790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349774 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768"} err="failed to get container status \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": rpc error: code = NotFound desc = could not find container \"b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768\": container with ID starting with b572166fa1f99a89485a4ab0cb0770b8e144304d89d9469be3521a9b6df45768 not found: ID does not exist" Oct 31 00:04:17.349843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349792 5943 scope.go:122] "RemoveContainer" containerID="b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98" Oct 31 00:04:17.350000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.349984 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98"} err="failed to get container status \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": rpc error: code = NotFound desc = could not find container \"b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98\": container with ID starting with b47be503782e428b287a6fdf1a6ea168f8ee6c8876440a3854b6ed625803dc98 not found: ID does not exist" Oct 31 00:04:17.350041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.350001 5943 scope.go:122] "RemoveContainer" containerID="a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d" Oct 31 00:04:17.350220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.350205 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d"} err="failed to get container status \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": rpc error: code = NotFound desc = could not find container \"a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d\": container with ID starting with a9e87ea1f406fafdbf4d4ba57d7d72be13d2b02bce60d88156b5baf39cede16d not found: ID does not exist" Oct 31 00:04:17.449310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.449278 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:04:17.499071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499037 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:04:17.499484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499468 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:04:17.499758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499747 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="prometheus" Oct 31 00:04:17.499795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499759 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="prometheus" Oct 31 00:04:17.499795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499774 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="thanos-sidecar" Oct 31 00:04:17.499795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499780 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="thanos-sidecar" Oct 31 00:04:17.499795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499788 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-web" Oct 31 00:04:17.499795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499794 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-web" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499803 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="config-reloader" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499807 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="config-reloader" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499817 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="init-config-reloader" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499822 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="init-config-reloader" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499828 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-thanos" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499833 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-thanos" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499840 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499844 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499905 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="prometheus" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499912 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="thanos-sidecar" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499918 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-web" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499925 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="config-reloader" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499931 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="init-config-reloader" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499935 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy-thanos" Oct 31 00:04:17.499973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.499940 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" containerName="kube-rbac-proxy" Oct 31 00:04:17.523762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.523737 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560318 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvrh\" (UniqueName: \"kubernetes.io/projected/487a23e3-9048-48fb-be8f-f29fcbaf8de4-kube-api-access-7mvrh\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560353 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560372 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-config\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560439 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/487a23e3-9048-48fb-be8f-f29fcbaf8de4-config-out\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560476 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560531 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560582 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560640 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560665 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560686 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/487a23e3-9048-48fb-be8f-f29fcbaf8de4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560701 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560813 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.560875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560853 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560904 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-web-config\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560952 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.560977 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.561023 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.561042 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.561062 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.561079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.561077 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.576807 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.576779 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:04:17.576943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.576917 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:04:17.576943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.576917 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:04:17.577086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577072 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:04:17.577148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577125 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:04:17.577245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577229 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:04:17.577520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577506 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:04:17.577612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577522 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:04:17.577612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577578 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:04:17.577612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577581 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:04:17.577612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577588 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:04:17.577813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577661 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lb62x\"" Oct 31 00:04:17.577813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577751 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:04:17.577912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.577890 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-adcr02j2qj3ro\"" Oct 31 00:04:17.627792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.627763 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:04:17.628005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.627878 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:04:17.661569 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661528 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.661569 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661572 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.661792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661692 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.661792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661728 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/487a23e3-9048-48fb-be8f-f29fcbaf8de4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.661792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661752 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.661792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661788 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661862 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661920 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-web-config\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661957 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.661986 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662026 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662052 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662088 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662109 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662116 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662165 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvrh\" (UniqueName: \"kubernetes.io/projected/487a23e3-9048-48fb-be8f-f29fcbaf8de4-kube-api-access-7mvrh\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662224 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662256 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-config\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662293 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/487a23e3-9048-48fb-be8f-f29fcbaf8de4-config-out\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662323 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662355 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662713 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.662817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.662797 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.663414 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.663105 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.663414 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.663121 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.663661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.663632 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.664737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.664691 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.666735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.666203 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.666735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.666633 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.667071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.667033 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-web-config\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.667071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.667028 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-config\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.667230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.667113 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/487a23e3-9048-48fb-be8f-f29fcbaf8de4-config-out\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.667489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.667467 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.667784 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.667760 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/487a23e3-9048-48fb-be8f-f29fcbaf8de4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.667873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.667828 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.668105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.668084 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.668144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.668129 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.668574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.668554 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/487a23e3-9048-48fb-be8f-f29fcbaf8de4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.668654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.668637 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/487a23e3-9048-48fb-be8f-f29fcbaf8de4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.724311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.724276 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvrh\" (UniqueName: \"kubernetes.io/projected/487a23e3-9048-48fb-be8f-f29fcbaf8de4-kube-api-access-7mvrh\") pod \"prometheus-k8s-0\" (UID: \"487a23e3-9048-48fb-be8f-f29fcbaf8de4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:17.754992 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.754954 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:04:17.943660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:17.943588 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:18.038797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:18.038764 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5616844-000c-4b6c-8150-e7421eb3146c" path="/var/lib/kubelet/pods/b5616844-000c-4b6c-8150-e7421eb3146c/volumes" Oct 31 00:04:18.442654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:18.442617 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:04:18.472585 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:04:18.472551 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487a23e3_9048_48fb_be8f_f29fcbaf8de4.slice/crio-3b4af919645ddf1a66b49cba765b230c07cae355a474393243331ded3ad70a69 WatchSource:0}: Error finding container 3b4af919645ddf1a66b49cba765b230c07cae355a474393243331ded3ad70a69: Status 404 returned error can't find the container with id 3b4af919645ddf1a66b49cba765b230c07cae355a474393243331ded3ad70a69 Oct 31 00:04:19.294390 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:19.294358 5943 generic.go:356] "Generic (PLEG): container finished" podID="487a23e3-9048-48fb-be8f-f29fcbaf8de4" containerID="42d80541d0762376dbbabd57fe4c6cbbfd7b81df81a5adb107d977d11324136e" exitCode=0 Oct 31 00:04:19.294649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:19.294447 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerDied","Data":"42d80541d0762376dbbabd57fe4c6cbbfd7b81df81a5adb107d977d11324136e"} Oct 31 00:04:19.294649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:19.294511 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"3b4af919645ddf1a66b49cba765b230c07cae355a474393243331ded3ad70a69"} Oct 31 00:04:20.305679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.305636 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"18d9d5c115f0568012b52deb0ce47318b093b959eb5c26d0f3aca0e35cabea79"} Oct 31 00:04:20.306119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.305725 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"a94060e696b51fff558f6f58e851710903e3a0591f598b81f52895c2401696ca"} Oct 31 00:04:20.306119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.305753 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"ed824624f9745ecc4242a281174776f370ed026fcaf42118f67f6c8187eb88aa"} Oct 31 00:04:20.306119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.305779 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"6c21fbc513ac0955fb591f84075ccc4e9d3d5e36533139840e2a09d109dd810b"} Oct 31 00:04:20.306119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.305802 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"b788a03a36877e9efe7c8d1d5d2efc78b5a92480778b4170a0bba4e573166e6e"} Oct 31 00:04:20.306119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.305825 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"487a23e3-9048-48fb-be8f-f29fcbaf8de4","Type":"ContainerStarted","Data":"f956dfe3da109ec81a39a7eaafea57a625c0e627c3599ff1b0aaae1232a5cb67"} Oct 31 00:04:20.437936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:20.437886 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.421669192 podStartE2EDuration="3.437871148s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.201956ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:20.436877478 +0000 UTC m=+40.530167720" watchObservedRunningTime="2026-10-31 00:04:20.437871148 +0000 UTC m=+40.531161313" Oct 31 00:04:22.944039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:22.944001 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:26.413993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:26.413960 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:26.558948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:26.558911 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:27.475195 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:27.475138 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:04:37.976289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:37.976249 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:42.600818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:42.600777 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:42.617207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:42.617162 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:42.961027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:04:42.960948 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:04:57.478396 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:04:57.478358 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:05:27.474811 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:05:27.474772 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:05:38.327075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:05:38.327028 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:41.506826 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:05:41.506785 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:41.701630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:05:41.701597 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:51.604685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:05:51.604646 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:52.374992 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:05:52.374938 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:57.475742 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:05:57.475694 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:06:06.517610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:06:06.517570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:27.474735 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:06:27.474684 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:06:33.677220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:06:33.677165 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:38.682295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:06:38.682261 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:57.475310 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:06:57.475090 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:07:25.958917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:07:25.958869 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:27.268353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:07:27.268303 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:27.474796 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:07:27.474651 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:07:39.036667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:07:39.036631 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:39.381406 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:07:39.381312 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:39.523874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:07:39.523833 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:57.475425 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:07:57.475386 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:08:27.475389 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:08:27.475346 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:08:39.402701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:08:39.402658 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:57.474948 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:08:57.474914 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:09:13.879113 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:13.879072 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:14.179435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:14.179346 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:15.980456 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:15.980419 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:16.284470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:16.284438 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:17.739293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:17.739253 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:21.949168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:21.949138 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:22.176603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:22.176570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:23.573069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.573030 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:23.768636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.768602 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk"] Oct 31 00:09:23.770798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.770777 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.781673 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.781651 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:23.856166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.856070 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:09:23.856166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.856091 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:09:23.856166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.856091 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-gggv4\"" Oct 31 00:09:23.856166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.856147 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:09:23.873330 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.873305 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk"] Oct 31 00:09:23.928519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.928485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:23.932587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932565 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.932715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932608 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.932715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932635 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.932827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932760 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fshn\" (UniqueName: \"kubernetes.io/projected/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-kube-api-access-5fshn\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.932890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932830 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.932944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932901 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.932981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932958 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.933034 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.932984 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:23.933082 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:23.933042 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033631 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033597 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033636 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033656 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033676 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033697 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033731 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033761 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.033836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033803 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fshn\" (UniqueName: \"kubernetes.io/projected/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-kube-api-access-5fshn\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.034237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.033851 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.034237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.034119 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.034237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.034151 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.034237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.034221 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.034438 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.034356 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.034842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.034816 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.036427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.036400 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.036535 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.036431 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.095070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.095036 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.095357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.095333 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fshn\" (UniqueName: \"kubernetes.io/projected/36de5579-6d9a-4a29-b77f-2d417d9f7dc5-kube-api-access-5fshn\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk\" (UID: \"36de5579-6d9a-4a29-b77f-2d417d9f7dc5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.382052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.382011 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:24.806207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.806151 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk"] Oct 31 00:09:24.809642 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:09:24.809614 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36de5579_6d9a_4a29_b77f_2d417d9f7dc5.slice/crio-7cee7829dc6eb8fb76c6914b1a3943e1e7d6a9b1382ab3fba8e5951890c91db7 WatchSource:0}: Error finding container 7cee7829dc6eb8fb76c6914b1a3943e1e7d6a9b1382ab3fba8e5951890c91db7: Status 404 returned error can't find the container with id 7cee7829dc6eb8fb76c6914b1a3943e1e7d6a9b1382ab3fba8e5951890c91db7 Oct 31 00:09:24.811685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:24.811668 5943 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:09:25.454173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.454135 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" event={"ID":"36de5579-6d9a-4a29-b77f-2d417d9f7dc5","Type":"ContainerStarted","Data":"7cee7829dc6eb8fb76c6914b1a3943e1e7d6a9b1382ab3fba8e5951890c91db7"} Oct 31 00:09:25.669130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.669096 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp"] Oct 31 00:09:25.671865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.671846 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.718547 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.718456 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz"] Oct 31 00:09:25.720892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.720869 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.767356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.767330 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:09:25.767356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.767328 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kb7dw\"" Oct 31 00:09:25.767356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.767357 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:09:25.776858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.776825 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp"] Oct 31 00:09:25.827500 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.827471 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-mwzmv\"" Oct 31 00:09:25.836125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.836091 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz"] Oct 31 00:09:25.849312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.849280 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfp8\" (UniqueName: \"kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.849473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.849332 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.849473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.849411 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rsh\" (UniqueName: \"kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.849473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.849447 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.849590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.849570 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.849626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.849604 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.950844 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.950798 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.951046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.950859 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rsh\" (UniqueName: \"kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.951046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.950891 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.951046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.950969 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.951046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.950997 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.951046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.951030 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfp8\" (UniqueName: \"kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.951363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.951347 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:25.951426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.951347 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.951426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.951381 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:25.951426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:25.951398 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:26.011961 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.011930 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfp8\" (UniqueName: \"kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:26.012158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.012129 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rsh\" (UniqueName: \"kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:26.033868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.033837 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:26.284990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.284916 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:26.404330 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.404299 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz"] Oct 31 00:09:26.407412 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:09:26.407381 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8233110c_de02_4b3e_8a17_bad071712a87.slice/crio-9f343ea801a56f7a333763825ef68b2cf890f59407f91f96686dd54654100b7d WatchSource:0}: Error finding container 9f343ea801a56f7a333763825ef68b2cf890f59407f91f96686dd54654100b7d: Status 404 returned error can't find the container with id 9f343ea801a56f7a333763825ef68b2cf890f59407f91f96686dd54654100b7d Oct 31 00:09:26.459687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.459650 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" event={"ID":"8233110c-de02-4b3e-8a17-bad071712a87","Type":"ContainerStarted","Data":"9f343ea801a56f7a333763825ef68b2cf890f59407f91f96686dd54654100b7d"} Oct 31 00:09:26.658159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:26.658077 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp"] Oct 31 00:09:26.664710 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:09:26.664678 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b990c2_741c_470e_91bd_2ee0c79836a1.slice/crio-af7d83c7c9bb5583e3cf78a3f30c822283d2ac44118159d2b2e4f8504fa8fe4c WatchSource:0}: Error finding container af7d83c7c9bb5583e3cf78a3f30c822283d2ac44118159d2b2e4f8504fa8fe4c: Status 404 returned error can't find the container with id af7d83c7c9bb5583e3cf78a3f30c822283d2ac44118159d2b2e4f8504fa8fe4c Oct 31 00:09:27.466722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:27.466685 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" event={"ID":"93b990c2-741c-470e-91bd-2ee0c79836a1","Type":"ContainerStarted","Data":"af7d83c7c9bb5583e3cf78a3f30c822283d2ac44118159d2b2e4f8504fa8fe4c"} Oct 31 00:09:27.480325 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:09:27.480283 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:09:28.112837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:28.112804 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:28.836410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:28.836372 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:09:28.836849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:28.836462 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:09:28.836849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:28.836502 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:09:29.481252 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:29.480589 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" event={"ID":"36de5579-6d9a-4a29-b77f-2d417d9f7dc5","Type":"ContainerStarted","Data":"f69c552142cd45859c04cc63a3a559f0f008c7f3412092e04bec40d99e3d1177"} Oct 31 00:09:29.650875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:29.650812 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" podStartSLOduration=2.6273349660000003 podStartE2EDuration="6.650793313s" totalImagesPullingTime="4.023458347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:29.650121836 +0000 UTC m=+349.743412048" watchObservedRunningTime="2026-10-31 00:09:29.650793313 +0000 UTC m=+349.744083498" Oct 31 00:09:30.382629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:30.382510 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:30.389049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:30.389017 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:30.486540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:30.486508 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:30.487574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:30.487520 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk" Oct 31 00:09:33.503285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:33.503255 5943 generic.go:356] "Generic (PLEG): container finished" podID="8233110c-de02-4b3e-8a17-bad071712a87" containerID="bd9968d0fcd2d81938bcdccde589440ca25724bf4891453177175c3a790780f3" exitCode=0 Oct 31 00:09:33.503762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:33.503309 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" event={"ID":"8233110c-de02-4b3e-8a17-bad071712a87","Type":"ContainerDied","Data":"bd9968d0fcd2d81938bcdccde589440ca25724bf4891453177175c3a790780f3"} Oct 31 00:09:34.511662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:34.511618 5943 generic.go:356] "Generic (PLEG): container finished" podID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerID="9b2cac5726f98827c37f32a3477cfbd67399cfa785aec5e489770e0a38ac0319" exitCode=0 Oct 31 00:09:34.512123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:34.511698 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" event={"ID":"93b990c2-741c-470e-91bd-2ee0c79836a1","Type":"ContainerDied","Data":"9b2cac5726f98827c37f32a3477cfbd67399cfa785aec5e489770e0a38ac0319"} Oct 31 00:09:35.523132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:35.523101 5943 generic.go:356] "Generic (PLEG): container finished" podID="8233110c-de02-4b3e-8a17-bad071712a87" containerID="3d7d26bb852b19b4dfd21f40429f6014afdb492d8debd6046585f39b9c66e39f" exitCode=0 Oct 31 00:09:35.523545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:35.523217 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" event={"ID":"8233110c-de02-4b3e-8a17-bad071712a87","Type":"ContainerDied","Data":"3d7d26bb852b19b4dfd21f40429f6014afdb492d8debd6046585f39b9c66e39f"} Oct 31 00:09:36.530620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:36.530578 5943 generic.go:356] "Generic (PLEG): container finished" podID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerID="46c56ae0985e38cfc6e2ac7d7000e65101c3130ff85958dfb9c2abef5d75e753" exitCode=0 Oct 31 00:09:36.531054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:36.530663 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" event={"ID":"93b990c2-741c-470e-91bd-2ee0c79836a1","Type":"ContainerDied","Data":"46c56ae0985e38cfc6e2ac7d7000e65101c3130ff85958dfb9c2abef5d75e753"} Oct 31 00:09:39.753417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:39.753266 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:43.570284 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:43.570249 5943 generic.go:356] "Generic (PLEG): container finished" podID="8233110c-de02-4b3e-8a17-bad071712a87" containerID="eb74ead556976600dd582962cdb47bd69dabac56834cc076ea34c27e11a03bb1" exitCode=0 Oct 31 00:09:43.570743 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:43.570333 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" event={"ID":"8233110c-de02-4b3e-8a17-bad071712a87","Type":"ContainerDied","Data":"eb74ead556976600dd582962cdb47bd69dabac56834cc076ea34c27e11a03bb1"} Oct 31 00:09:44.577121 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.577028 5943 generic.go:356] "Generic (PLEG): container finished" podID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerID="c420834e0181966a80074ccd467826fe83ecff4e07122679bf67b78fc582e3e9" exitCode=0 Oct 31 00:09:44.577121 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.577069 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" event={"ID":"93b990c2-741c-470e-91bd-2ee0c79836a1","Type":"ContainerDied","Data":"c420834e0181966a80074ccd467826fe83ecff4e07122679bf67b78fc582e3e9"} Oct 31 00:09:44.706621 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.706594 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:44.735346 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.735316 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle\") pod \"8233110c-de02-4b3e-8a17-bad071712a87\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " Oct 31 00:09:44.735546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.735359 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util\") pod \"8233110c-de02-4b3e-8a17-bad071712a87\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " Oct 31 00:09:44.735546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.735418 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8\" (UniqueName: \"kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8\") pod \"8233110c-de02-4b3e-8a17-bad071712a87\" (UID: \"8233110c-de02-4b3e-8a17-bad071712a87\") " Oct 31 00:09:44.735800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.735771 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle" pod "8233110c-de02-4b3e-8a17-bad071712a87" (UID: "8233110c-de02-4b3e-8a17-bad071712a87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:44.737799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.737776 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8" pod "8233110c-de02-4b3e-8a17-bad071712a87" (UID: "8233110c-de02-4b3e-8a17-bad071712a87"). InnerVolumeSpecName "kube-api-access-zkfp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:44.739347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.739326 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util" pod "8233110c-de02-4b3e-8a17-bad071712a87" (UID: "8233110c-de02-4b3e-8a17-bad071712a87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:44.837058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.836985 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfp8\" (UniqueName: \"kubernetes.io/projected/8233110c-de02-4b3e-8a17-bad071712a87-kube-api-access-zkfp8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:44.837058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.837014 5943 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-bundle\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:44.837058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:44.837024 5943 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8233110c-de02-4b3e-8a17-bad071712a87-util\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:45.588345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.588306 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" Oct 31 00:09:45.588345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.588325 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hbzfz" event={"ID":"8233110c-de02-4b3e-8a17-bad071712a87","Type":"ContainerDied","Data":"9f343ea801a56f7a333763825ef68b2cf890f59407f91f96686dd54654100b7d"} Oct 31 00:09:45.588904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.588396 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f343ea801a56f7a333763825ef68b2cf890f59407f91f96686dd54654100b7d" Oct 31 00:09:45.718282 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.718249 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:45.845465 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.845366 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util\") pod \"93b990c2-741c-470e-91bd-2ee0c79836a1\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " Oct 31 00:09:45.845465 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.845420 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh\" (UniqueName: \"kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh\") pod \"93b990c2-741c-470e-91bd-2ee0c79836a1\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " Oct 31 00:09:45.845657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.845474 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle\") pod \"93b990c2-741c-470e-91bd-2ee0c79836a1\" (UID: \"93b990c2-741c-470e-91bd-2ee0c79836a1\") " Oct 31 00:09:45.846271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.846168 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle" pod "93b990c2-741c-470e-91bd-2ee0c79836a1" (UID: "93b990c2-741c-470e-91bd-2ee0c79836a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:45.847836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.847805 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh" pod "93b990c2-741c-470e-91bd-2ee0c79836a1" (UID: "93b990c2-741c-470e-91bd-2ee0c79836a1"). InnerVolumeSpecName "kube-api-access-h7rsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:45.850820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.850793 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util" pod "93b990c2-741c-470e-91bd-2ee0c79836a1" (UID: "93b990c2-741c-470e-91bd-2ee0c79836a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:45.946316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.946277 5943 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-util\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:45.946316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.946309 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7rsh\" (UniqueName: \"kubernetes.io/projected/93b990c2-741c-470e-91bd-2ee0c79836a1-kube-api-access-h7rsh\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:45.946316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:45.946323 5943 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93b990c2-741c-470e-91bd-2ee0c79836a1-bundle\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:46.594587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:46.594543 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" event={"ID":"93b990c2-741c-470e-91bd-2ee0c79836a1","Type":"ContainerDied","Data":"af7d83c7c9bb5583e3cf78a3f30c822283d2ac44118159d2b2e4f8504fa8fe4c"} Oct 31 00:09:46.594587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:46.594582 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abhsjp" Oct 31 00:09:46.595150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:09:46.594598 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7d83c7c9bb5583e3cf78a3f30c822283d2ac44118159d2b2e4f8504fa8fe4c" Oct 31 00:09:57.475802 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:09:57.475717 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:10:04.283919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:04.283875 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:04.475345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:04.475304 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:05.354097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.354065 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:05.637867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.637774 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4"] Oct 31 00:10:05.638438 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638417 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="util" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638440 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="util" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638466 5943 cpu_manager.go:414] "containerMap: removing container" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="extract" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638473 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="extract" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638487 5943 cpu_manager.go:414] "containerMap: removing container" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="util" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638493 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="util" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638508 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="extract" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638515 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="extract" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638525 5943 cpu_manager.go:414] "containerMap: removing container" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="pull" Oct 31 00:10:05.638546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638531 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="pull" Oct 31 00:10:05.638967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638568 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="pull" Oct 31 00:10:05.638967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638575 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="pull" Oct 31 00:10:05.638967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638663 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b990c2-741c-470e-91bd-2ee0c79836a1" containerName="extract" Oct 31 00:10:05.638967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.638678 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8233110c-de02-4b3e-8a17-bad071712a87" containerName="extract" Oct 31 00:10:05.640830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.640808 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.701980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.701951 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:10:05.702380 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.702367 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:10:05.707895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.707869 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:05.710990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.710973 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:10:05.736540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.736504 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ts4\" (UniqueName: \"kubernetes.io/projected/1af2e2ad-7002-47af-b057-eb0c6a3683fe-kube-api-access-h9ts4\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.736707 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.736628 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.736707 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.736679 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1af2e2ad-7002-47af-b057-eb0c6a3683fe-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.736821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.736722 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.736821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.736771 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.736904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.736865 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1af2e2ad-7002-47af-b057-eb0c6a3683fe-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.751436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.751405 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:10:05.751597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.751478 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:10:05.751597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.751526 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-25dgf\"" Oct 31 00:10:05.753099 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.753082 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4"] Oct 31 00:10:05.837711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.837669 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ts4\" (UniqueName: \"kubernetes.io/projected/1af2e2ad-7002-47af-b057-eb0c6a3683fe-kube-api-access-h9ts4\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.837925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.837727 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.837925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.837760 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1af2e2ad-7002-47af-b057-eb0c6a3683fe-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.837925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.837794 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.837925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.837811 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.837925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.837836 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1af2e2ad-7002-47af-b057-eb0c6a3683fe-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.838384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.838357 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1af2e2ad-7002-47af-b057-eb0c6a3683fe-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.840649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.840618 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.840776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.840618 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.840846 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.840780 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1af2e2ad-7002-47af-b057-eb0c6a3683fe-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.840891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.840879 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1af2e2ad-7002-47af-b057-eb0c6a3683fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.869859 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.869823 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:05.899980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.899913 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ts4\" (UniqueName: \"kubernetes.io/projected/1af2e2ad-7002-47af-b057-eb0c6a3683fe-kube-api-access-h9ts4\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-lj2c4\" (UID: \"1af2e2ad-7002-47af-b057-eb0c6a3683fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:05.953691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:05.953649 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:06.021360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:06.021328 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:06.487171 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:06.487034 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4"] Oct 31 00:10:06.489542 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:06.489511 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af2e2ad_7002_47af_b057_eb0c6a3683fe.slice/crio-ce5f53a81d71d33e1f6f3123aeec9e026d1f7706e2a41f4d2cea6f52432d2aa4 WatchSource:0}: Error finding container ce5f53a81d71d33e1f6f3123aeec9e026d1f7706e2a41f4d2cea6f52432d2aa4: Status 404 returned error can't find the container with id ce5f53a81d71d33e1f6f3123aeec9e026d1f7706e2a41f4d2cea6f52432d2aa4 Oct 31 00:10:06.561709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:06.561675 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:06.674487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:06.674456 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" event={"ID":"1af2e2ad-7002-47af-b057-eb0c6a3683fe","Type":"ContainerStarted","Data":"ce5f53a81d71d33e1f6f3123aeec9e026d1f7706e2a41f4d2cea6f52432d2aa4"} Oct 31 00:10:07.969271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:07.969239 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:08.173168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:08.173102 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:08.455856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:08.455823 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.701880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:10.701789 5943 generic.go:356] "Generic (PLEG): container finished" podID="1af2e2ad-7002-47af-b057-eb0c6a3683fe" containerID="933d17035767875ced3b17bfc216c8446bc2a9f5784bda6137cb222097131ee7" exitCode=0 Oct 31 00:10:10.702304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:10.701869 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" event={"ID":"1af2e2ad-7002-47af-b057-eb0c6a3683fe","Type":"ContainerDied","Data":"933d17035767875ced3b17bfc216c8446bc2a9f5784bda6137cb222097131ee7"} Oct 31 00:10:11.711498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:11.711452 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" event={"ID":"1af2e2ad-7002-47af-b057-eb0c6a3683fe","Type":"ContainerStarted","Data":"9b9f3cd3b5b66888156d50b7f97117dab40673ef3c4bbce5b420dfb9ca050298"} Oct 31 00:10:11.712012 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:11.711537 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:11.951486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:11.951371 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" podStartSLOduration=2.233001491 podStartE2EDuration="6.951356322s" totalImagesPullingTime="4.180140538s" totalInitContainerRuntime="538.214293ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.901853761 +0000 UTC m=+391.995143949" watchObservedRunningTime="2026-10-31 00:10:11.951356322 +0000 UTC m=+392.044646509" Oct 31 00:10:12.640072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:12.640034 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:13.520993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:13.520943 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:13.721971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:13.721936 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:13.872510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:13.872410 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:14.106426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:14.106391 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:16.285379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:16.285344 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:16.493733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:16.493695 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:17.019250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:17.019211 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:17.342300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:17.342210 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:17.536347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:17.536306 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:18.360763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.360729 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd"] Oct 31 00:10:18.364345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.364323 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.437534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.437498 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bk8g9\"" Oct 31 00:10:18.437717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.437685 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:10:18.437759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.437743 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:10:18.440649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.440623 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd"] Oct 31 00:10:18.463710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.463671 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nksvx\" (UniqueName: \"kubernetes.io/projected/184f38ef-5f48-45ae-9deb-b919c8e75e51-kube-api-access-nksvx\") pod \"cert-manager-webhook-5dbbbf75cc-2tmfd\" (UID: \"184f38ef-5f48-45ae-9deb-b919c8e75e51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.463710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.463717 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184f38ef-5f48-45ae-9deb-b919c8e75e51-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2tmfd\" (UID: \"184f38ef-5f48-45ae-9deb-b919c8e75e51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.565066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.565025 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nksvx\" (UniqueName: \"kubernetes.io/projected/184f38ef-5f48-45ae-9deb-b919c8e75e51-kube-api-access-nksvx\") pod \"cert-manager-webhook-5dbbbf75cc-2tmfd\" (UID: \"184f38ef-5f48-45ae-9deb-b919c8e75e51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.565066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.565069 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184f38ef-5f48-45ae-9deb-b919c8e75e51-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2tmfd\" (UID: \"184f38ef-5f48-45ae-9deb-b919c8e75e51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.640741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.640662 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nksvx\" (UniqueName: \"kubernetes.io/projected/184f38ef-5f48-45ae-9deb-b919c8e75e51-kube-api-access-nksvx\") pod \"cert-manager-webhook-5dbbbf75cc-2tmfd\" (UID: \"184f38ef-5f48-45ae-9deb-b919c8e75e51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.640972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.640799 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184f38ef-5f48-45ae-9deb-b919c8e75e51-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2tmfd\" (UID: \"184f38ef-5f48-45ae-9deb-b919c8e75e51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:18.693255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:18.693220 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:19.087376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.087345 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd"] Oct 31 00:10:19.091341 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:19.091313 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod184f38ef_5f48_45ae_9deb_b919c8e75e51.slice/crio-f0f243598c862404e553b539941449276907ed14240c03f5c7bd652d3a7aa6e3 WatchSource:0}: Error finding container f0f243598c862404e553b539941449276907ed14240c03f5c7bd652d3a7aa6e3: Status 404 returned error can't find the container with id f0f243598c862404e553b539941449276907ed14240c03f5c7bd652d3a7aa6e3 Oct 31 00:10:19.455396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.455299 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:19.747720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.747630 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" event={"ID":"184f38ef-5f48-45ae-9deb-b919c8e75e51","Type":"ContainerStarted","Data":"f0f243598c862404e553b539941449276907ed14240c03f5c7bd652d3a7aa6e3"} Oct 31 00:10:19.867628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.867597 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:19.876332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.876300 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4"] Oct 31 00:10:19.878589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.878569 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:19.930900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.930861 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gdb2m\"" Oct 31 00:10:19.977110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.977077 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmgp\" (UniqueName: \"kubernetes.io/projected/1f38835d-d2a3-4af2-86d4-13b5435f485c-kube-api-access-spmgp\") pod \"cert-manager-cainjector-5c57bbd4fc-5rmm4\" (UID: \"1f38835d-d2a3-4af2-86d4-13b5435f485c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:19.977366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.977133 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f38835d-d2a3-4af2-86d4-13b5435f485c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5rmm4\" (UID: \"1f38835d-d2a3-4af2-86d4-13b5435f485c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:19.981328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:19.981305 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4"] Oct 31 00:10:20.078551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.078514 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmgp\" (UniqueName: \"kubernetes.io/projected/1f38835d-d2a3-4af2-86d4-13b5435f485c-kube-api-access-spmgp\") pod \"cert-manager-cainjector-5c57bbd4fc-5rmm4\" (UID: \"1f38835d-d2a3-4af2-86d4-13b5435f485c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:20.078751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.078566 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f38835d-d2a3-4af2-86d4-13b5435f485c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5rmm4\" (UID: \"1f38835d-d2a3-4af2-86d4-13b5435f485c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:20.146261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.146232 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f38835d-d2a3-4af2-86d4-13b5435f485c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5rmm4\" (UID: \"1f38835d-d2a3-4af2-86d4-13b5435f485c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:20.146411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.146355 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmgp\" (UniqueName: \"kubernetes.io/projected/1f38835d-d2a3-4af2-86d4-13b5435f485c-kube-api-access-spmgp\") pod \"cert-manager-cainjector-5c57bbd4fc-5rmm4\" (UID: \"1f38835d-d2a3-4af2-86d4-13b5435f485c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:20.189387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.189348 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" Oct 31 00:10:20.559818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.559787 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4"] Oct 31 00:10:20.562965 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:20.562930 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f38835d_d2a3_4af2_86d4_13b5435f485c.slice/crio-484089c1342735bb8b740cb01f8018b16aae9a2423a7390c70696b045b5815d2 WatchSource:0}: Error finding container 484089c1342735bb8b740cb01f8018b16aae9a2423a7390c70696b045b5815d2: Status 404 returned error can't find the container with id 484089c1342735bb8b740cb01f8018b16aae9a2423a7390c70696b045b5815d2 Oct 31 00:10:20.752493 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:20.752457 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" event={"ID":"1f38835d-d2a3-4af2-86d4-13b5435f485c","Type":"ContainerStarted","Data":"484089c1342735bb8b740cb01f8018b16aae9a2423a7390c70696b045b5815d2"} Oct 31 00:10:22.720338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:22.720298 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-lj2c4" Oct 31 00:10:23.648203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:23.648142 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:24.673458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:24.673417 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:24.777337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:24.777298 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" event={"ID":"184f38ef-5f48-45ae-9deb-b919c8e75e51","Type":"ContainerStarted","Data":"1bcb525a2b0d89ac0f5b6db9207a7cf9dd3ff2291fad9f4494b52f709a39e391"} Oct 31 00:10:24.777528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:24.777365 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:24.907681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:24.907635 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" podStartSLOduration=1.8258529000000001 podStartE2EDuration="6.907623412s" totalImagesPullingTime="5.081770512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:24.906446852 +0000 UTC m=+404.999737060" watchObservedRunningTime="2026-10-31 00:10:24.907623412 +0000 UTC m=+405.000913596" Oct 31 00:10:25.694168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:25.694128 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.785371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:25.785334 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" event={"ID":"1f38835d-d2a3-4af2-86d4-13b5435f485c","Type":"ContainerStarted","Data":"11a9655e9a7b74a3220c6b184817441dc58c4de7b1d0929998636a4bcb55895a"} Oct 31 00:10:25.920799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:25.920749 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5rmm4" podStartSLOduration=2.606597471 podStartE2EDuration="6.920733782s" totalImagesPullingTime="4.314136311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:25.920002988 +0000 UTC m=+406.013293193" watchObservedRunningTime="2026-10-31 00:10:25.920733782 +0000 UTC m=+406.014023969" Oct 31 00:10:26.455481 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:26.455446 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:26.616890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:26.616853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:27.475173 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:10:27.475134 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:10:27.576288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:27.576247 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:30.654420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:30.654379 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:30.788664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:30.788630 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2tmfd" Oct 31 00:10:31.086626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:31.086591 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:32.207761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:32.207725 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:32.414100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:32.414056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:33.449740 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:33.449703 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:33.824235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:33.824166 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:33.990238 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:33.990203 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:35.101760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.101728 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx"] Oct 31 00:10:35.107118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.107081 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:35.214899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.214864 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:10:35.214899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.214863 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ws659\"" Oct 31 00:10:35.215120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.215020 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:10:35.216059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.216039 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx"] Oct 31 00:10:35.230653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.230621 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkqv\" (UniqueName: \"kubernetes.io/projected/4640a92d-1063-4a9f-afbb-6221ba3bc084-kube-api-access-grkqv\") pod \"limitador-operator-controller-manager-5778bb4449-hlvcx\" (UID: \"4640a92d-1063-4a9f-afbb-6221ba3bc084\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:35.331805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.331765 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkqv\" (UniqueName: \"kubernetes.io/projected/4640a92d-1063-4a9f-afbb-6221ba3bc084-kube-api-access-grkqv\") pod \"limitador-operator-controller-manager-5778bb4449-hlvcx\" (UID: \"4640a92d-1063-4a9f-afbb-6221ba3bc084\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:35.392691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.392615 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkqv\" (UniqueName: \"kubernetes.io/projected/4640a92d-1063-4a9f-afbb-6221ba3bc084-kube-api-access-grkqv\") pod \"limitador-operator-controller-manager-5778bb4449-hlvcx\" (UID: \"4640a92d-1063-4a9f-afbb-6221ba3bc084\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:35.420472 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.420440 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:35.650940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.650847 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:35.796844 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.796814 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx"] Oct 31 00:10:35.800543 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:35.800517 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4640a92d_1063_4a9f_afbb_6221ba3bc084.slice/crio-c8278981ab22167079c9f5bfd37ec7ae65c2d8628e74be995bfda44384721f2e WatchSource:0}: Error finding container c8278981ab22167079c9f5bfd37ec7ae65c2d8628e74be995bfda44384721f2e: Status 404 returned error can't find the container with id c8278981ab22167079c9f5bfd37ec7ae65c2d8628e74be995bfda44384721f2e Oct 31 00:10:35.827567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.827541 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" event={"ID":"4640a92d-1063-4a9f-afbb-6221ba3bc084","Type":"ContainerStarted","Data":"c8278981ab22167079c9f5bfd37ec7ae65c2d8628e74be995bfda44384721f2e"} Oct 31 00:10:35.905254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:35.905143 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:36.794025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:36.793992 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:36.941670 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:36.941630 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:37.474362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:37.474329 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:38.742359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:38.742329 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:38.907104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:38.907065 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:39.029151 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.029110 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx"] Oct 31 00:10:39.032764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.032745 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.088198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.088124 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mk2zv\"" Oct 31 00:10:39.088198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.088127 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:10:39.088428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.088245 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 00:10:39.089362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.089335 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:39.096834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.096808 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx"] Oct 31 00:10:39.169514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.169455 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlv5n\" (UniqueName: \"kubernetes.io/projected/8a21f85d-c34b-4149-a0c1-2ba949b1b120-kube-api-access-rlv5n\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.169683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.169565 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a21f85d-c34b-4149-a0c1-2ba949b1b120-tmp\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.169683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.169596 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a21f85d-c34b-4149-a0c1-2ba949b1b120-tls-certs\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.270511 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.270471 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlv5n\" (UniqueName: \"kubernetes.io/projected/8a21f85d-c34b-4149-a0c1-2ba949b1b120-kube-api-access-rlv5n\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.270710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.270525 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a21f85d-c34b-4149-a0c1-2ba949b1b120-tmp\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.270710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.270559 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a21f85d-c34b-4149-a0c1-2ba949b1b120-tls-certs\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.273415 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.273385 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a21f85d-c34b-4149-a0c1-2ba949b1b120-tmp\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.273696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.273673 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a21f85d-c34b-4149-a0c1-2ba949b1b120-tls-certs\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.331819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.331743 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlv5n\" (UniqueName: \"kubernetes.io/projected/8a21f85d-c34b-4149-a0c1-2ba949b1b120-kube-api-access-rlv5n\") pod \"kube-auth-proxy-5cb564c48d-j7qlx\" (UID: \"8a21f85d-c34b-4149-a0c1-2ba949b1b120\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.347614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.347583 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" Oct 31 00:10:39.470742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.470692 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:39.829279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.829248 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx"] Oct 31 00:10:39.829672 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:39.829635 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a21f85d_c34b_4149_a0c1_2ba949b1b120.slice/crio-8bc370f1e1e7d5320d44b2e6b5f943ebbb8897b953e1d60673ec763f6ee682ad WatchSource:0}: Error finding container 8bc370f1e1e7d5320d44b2e6b5f943ebbb8897b953e1d60673ec763f6ee682ad: Status 404 returned error can't find the container with id 8bc370f1e1e7d5320d44b2e6b5f943ebbb8897b953e1d60673ec763f6ee682ad Oct 31 00:10:39.848521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.848487 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" event={"ID":"8a21f85d-c34b-4149-a0c1-2ba949b1b120","Type":"ContainerStarted","Data":"8bc370f1e1e7d5320d44b2e6b5f943ebbb8897b953e1d60673ec763f6ee682ad"} Oct 31 00:10:39.850099 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.850077 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" event={"ID":"4640a92d-1063-4a9f-afbb-6221ba3bc084","Type":"ContainerStarted","Data":"e516b02fb66ac58c2c5671d41bdbbafb6fb5dbd1ae8a5fe7b1532b37728f2809"} Oct 31 00:10:39.850264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.850243 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:39.926449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.926409 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:39.978032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:39.977972 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" podStartSLOduration=1.393558231 podStartE2EDuration="4.977955504s" totalImagesPullingTime="3.584397273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:39.976892715 +0000 UTC m=+420.070182924" watchObservedRunningTime="2026-10-31 00:10:39.977955504 +0000 UTC m=+420.071245689" Oct 31 00:10:40.095775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:40.095689 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:40.250919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:40.250884 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:40.747781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:40.747744 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:40.939925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:40.939887 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:41.096233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:41.096197 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:43.037634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:43.037597 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:10:43.563026 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:43.562989 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:43.872615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:43.872537 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" event={"ID":"8a21f85d-c34b-4149-a0c1-2ba949b1b120","Type":"ContainerStarted","Data":"2225a0119dc48a939b3f84961b22d895a017bd1eebb8b4484f91764cd5fdf1d3"} Oct 31 00:10:43.927474 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:43.927441 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:44.081120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:44.081081 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:45.195356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.195294 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-j7qlx" podStartSLOduration=4.041756552 podStartE2EDuration="7.195275284s" totalImagesPullingTime="3.153518732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:43.996342272 +0000 UTC m=+424.089632478" watchObservedRunningTime="2026-10-31 00:10:45.195275284 +0000 UTC m=+425.288565524" Oct 31 00:10:45.195759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.195506 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4b2zg"] Oct 31 00:10:45.200833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.200816 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:45.256878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.256845 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4b2zg"] Oct 31 00:10:45.305093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.305056 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7p7cn\"" Oct 31 00:10:45.327494 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.327462 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkznf\" (UniqueName: \"kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf\") pod \"authorino-57f7f8659f-4b2zg\" (UID: \"26611109-1a21-49f2-9b56-d2004f8cf05a\") " pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:45.348217 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.348156 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:45.428239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.428167 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkznf\" (UniqueName: \"kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf\") pod \"authorino-57f7f8659f-4b2zg\" (UID: \"26611109-1a21-49f2-9b56-d2004f8cf05a\") " pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:45.490107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.490021 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:45.491370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.491345 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkznf\" (UniqueName: \"kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf\") pod \"authorino-57f7f8659f-4b2zg\" (UID: \"26611109-1a21-49f2-9b56-d2004f8cf05a\") " pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:45.511204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.511153 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:45.879484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:45.879452 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4b2zg"] Oct 31 00:10:45.882703 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:45.882677 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26611109_1a21_49f2_9b56_d2004f8cf05a.slice/crio-9c8ffb2f44a7ac06fa9fa534608097000a8f75c19272efbae6af58c6d9f8e473 WatchSource:0}: Error finding container 9c8ffb2f44a7ac06fa9fa534608097000a8f75c19272efbae6af58c6d9f8e473: Status 404 returned error can't find the container with id 9c8ffb2f44a7ac06fa9fa534608097000a8f75c19272efbae6af58c6d9f8e473 Oct 31 00:10:46.682766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:46.682715 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:46.887601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:46.887566 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" event={"ID":"26611109-1a21-49f2-9b56-d2004f8cf05a","Type":"ContainerStarted","Data":"9c8ffb2f44a7ac06fa9fa534608097000a8f75c19272efbae6af58c6d9f8e473"} Oct 31 00:10:48.039971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.039938 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:10:48.042416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.042398 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.051340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.050901 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvffl\" (UniqueName: \"kubernetes.io/projected/7dc29b08-40a6-4473-90e8-48ac49e661d3-kube-api-access-fvffl\") pod \"limitador-limitador-7c9b87d574-d2fw9\" (UID: \"7dc29b08-40a6-4473-90e8-48ac49e661d3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.051340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.050945 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7dc29b08-40a6-4473-90e8-48ac49e661d3-config-file\") pod \"limitador-limitador-7c9b87d574-d2fw9\" (UID: \"7dc29b08-40a6-4473-90e8-48ac49e661d3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.094438 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.094411 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:10:48.145233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.144128 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kjrmg\"" Oct 31 00:10:48.146061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.146022 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:10:48.151681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.151657 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvffl\" (UniqueName: \"kubernetes.io/projected/7dc29b08-40a6-4473-90e8-48ac49e661d3-kube-api-access-fvffl\") pod \"limitador-limitador-7c9b87d574-d2fw9\" (UID: \"7dc29b08-40a6-4473-90e8-48ac49e661d3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.151809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.151695 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7dc29b08-40a6-4473-90e8-48ac49e661d3-config-file\") pod \"limitador-limitador-7c9b87d574-d2fw9\" (UID: \"7dc29b08-40a6-4473-90e8-48ac49e661d3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.152410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.152383 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7dc29b08-40a6-4473-90e8-48ac49e661d3-config-file\") pod \"limitador-limitador-7c9b87d574-d2fw9\" (UID: \"7dc29b08-40a6-4473-90e8-48ac49e661d3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.165967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.165876 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:48.213980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.213948 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvffl\" (UniqueName: \"kubernetes.io/projected/7dc29b08-40a6-4473-90e8-48ac49e661d3-kube-api-access-fvffl\") pod \"limitador-limitador-7c9b87d574-d2fw9\" (UID: \"7dc29b08-40a6-4473-90e8-48ac49e661d3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.339096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.339015 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:10:48.356842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.356807 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:48.444461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.444427 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:48.734782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.734402 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:10:48.737150 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:10:48.737109 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc29b08_40a6_4473_90e8_48ac49e661d3.slice/crio-b2abcb245fd3eec9fde1167f949db60abc00fda153bc000fe0c494ccf46637a2 WatchSource:0}: Error finding container b2abcb245fd3eec9fde1167f949db60abc00fda153bc000fe0c494ccf46637a2: Status 404 returned error can't find the container with id b2abcb245fd3eec9fde1167f949db60abc00fda153bc000fe0c494ccf46637a2 Oct 31 00:10:48.898198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:48.898138 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" event={"ID":"7dc29b08-40a6-4473-90e8-48ac49e661d3","Type":"ContainerStarted","Data":"b2abcb245fd3eec9fde1167f949db60abc00fda153bc000fe0c494ccf46637a2"} Oct 31 00:10:49.903845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:49.903796 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" event={"ID":"26611109-1a21-49f2-9b56-d2004f8cf05a","Type":"ContainerStarted","Data":"9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9"} Oct 31 00:10:49.955612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:49.955572 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:50.018695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:50.018634 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" podStartSLOduration=1.846017191 podStartE2EDuration="5.018618606s" totalImagesPullingTime="3.172601415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:50.017601811 +0000 UTC m=+430.110892002" watchObservedRunningTime="2026-10-31 00:10:50.018618606 +0000 UTC m=+430.111908794" Oct 31 00:10:50.156642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:50.156548 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:50.858404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:50.858372 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hlvcx" Oct 31 00:10:50.909590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:50.909549 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" event={"ID":"7dc29b08-40a6-4473-90e8-48ac49e661d3","Type":"ContainerStarted","Data":"151349a16fddd19b5ef9888c4b93c486b436c09a4f0ad3d0dfba2e4a6581d453"} Oct 31 00:10:51.136531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:51.136464 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" podStartSLOduration=2.148554314 podStartE2EDuration="4.136445961s" totalImagesPullingTime="1.987891647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:51.135628732 +0000 UTC m=+431.228918942" watchObservedRunningTime="2026-10-31 00:10:51.136445961 +0000 UTC m=+431.229736126" Oct 31 00:10:51.914918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:51.914888 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:10:52.867471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:52.867435 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:52.915015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:52.909287 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4b2zg"] Oct 31 00:10:52.915015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:52.909533 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" podUID="26611109-1a21-49f2-9b56-d2004f8cf05a" containerName="authorino" containerID="cri-o://9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9" gracePeriod=30 Oct 31 00:10:53.012603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.012572 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:53.160365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.160341 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:53.192167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.192135 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf\" (UniqueName: \"kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf\") pod \"26611109-1a21-49f2-9b56-d2004f8cf05a\" (UID: \"26611109-1a21-49f2-9b56-d2004f8cf05a\") " Oct 31 00:10:53.194448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.194423 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf" pod "26611109-1a21-49f2-9b56-d2004f8cf05a" (UID: "26611109-1a21-49f2-9b56-d2004f8cf05a"). InnerVolumeSpecName "kube-api-access-zkznf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:53.293589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.293557 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkznf\" (UniqueName: \"kubernetes.io/projected/26611109-1a21-49f2-9b56-d2004f8cf05a-kube-api-access-zkznf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.927233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.927196 5943 generic.go:356] "Generic (PLEG): container finished" podID="26611109-1a21-49f2-9b56-d2004f8cf05a" containerID="9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9" exitCode=0 Oct 31 00:10:53.927233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.927242 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" Oct 31 00:10:53.927723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.927282 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" event={"ID":"26611109-1a21-49f2-9b56-d2004f8cf05a","Type":"ContainerDied","Data":"9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9"} Oct 31 00:10:53.927723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.927320 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4b2zg" event={"ID":"26611109-1a21-49f2-9b56-d2004f8cf05a","Type":"ContainerDied","Data":"9c8ffb2f44a7ac06fa9fa534608097000a8f75c19272efbae6af58c6d9f8e473"} Oct 31 00:10:53.927723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.927337 5943 scope.go:122] "RemoveContainer" containerID="9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9" Oct 31 00:10:53.936230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.936215 5943 scope.go:122] "RemoveContainer" containerID="9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9" Oct 31 00:10:53.936498 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:10:53.936481 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9\": container with ID starting with 9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9 not found: ID does not exist" containerID="9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9" Oct 31 00:10:53.936544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:53.936507 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9"} err="failed to get container status \"9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9\": rpc error: code = NotFound desc = could not find container \"9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9\": container with ID starting with 9acc2d9f2c877ca520d7660f54666060e6186624036d7ffdc2343b51217171a9 not found: ID does not exist" Oct 31 00:10:54.089428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:54.089394 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4b2zg"] Oct 31 00:10:54.093797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:54.093772 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4b2zg"] Oct 31 00:10:55.333309 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:55.333273 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:55.491939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:55.491864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:56.039801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:56.039770 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26611109-1a21-49f2-9b56-d2004f8cf05a" path="/var/lib/kubelet/pods/26611109-1a21-49f2-9b56-d2004f8cf05a/volumes" Oct 31 00:10:57.179151 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:57.179115 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:57.325087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:57.325052 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:57.468824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:10:57.468735 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:57.475084 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:10:57.475057 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:11:01.992777 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:01.992732 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:02.138660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:02.138620 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:02.921303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:02.921276 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-d2fw9" Oct 31 00:11:05.723931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.723898 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6"] Oct 31 00:11:05.724413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.724336 5943 cpu_manager.go:414] "containerMap: removing container" podUID="26611109-1a21-49f2-9b56-d2004f8cf05a" containerName="authorino" Oct 31 00:11:05.724413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.724348 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26611109-1a21-49f2-9b56-d2004f8cf05a" containerName="authorino" Oct 31 00:11:05.724413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.724403 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="26611109-1a21-49f2-9b56-d2004f8cf05a" containerName="authorino" Oct 31 00:11:05.729297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.729276 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.781481 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.781452 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:11:05.810720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.810687 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2884d99-17e0-486e-ba7e-fec1605f05f2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.810875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.810743 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2884d99-17e0-486e-ba7e-fec1605f05f2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.810875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.810758 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2884d99-17e0-486e-ba7e-fec1605f05f2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.810945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.810900 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5vg\" (UniqueName: \"kubernetes.io/projected/c2884d99-17e0-486e-ba7e-fec1605f05f2-kube-api-access-wr5vg\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.830937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.830912 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-l79h2\"" Oct 31 00:11:05.830937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.830943 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:11:05.912003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.911968 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5vg\" (UniqueName: \"kubernetes.io/projected/c2884d99-17e0-486e-ba7e-fec1605f05f2-kube-api-access-wr5vg\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.912172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.912019 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2884d99-17e0-486e-ba7e-fec1605f05f2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.912172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.912061 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2884d99-17e0-486e-ba7e-fec1605f05f2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.912172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.912075 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2884d99-17e0-486e-ba7e-fec1605f05f2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.912487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.912470 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2884d99-17e0-486e-ba7e-fec1605f05f2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.912525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.912508 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2884d99-17e0-486e-ba7e-fec1605f05f2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.930895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.930868 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 00:11:05.932269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.932254 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6"] Oct 31 00:11:05.932969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.932952 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2884d99-17e0-486e-ba7e-fec1605f05f2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:05.976527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:05.976452 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5vg\" (UniqueName: \"kubernetes.io/projected/c2884d99-17e0-486e-ba7e-fec1605f05f2-kube-api-access-wr5vg\") pod \"kserve-module-controller-manager-7566cf5fc8-nmcf6\" (UID: \"c2884d99-17e0-486e-ba7e-fec1605f05f2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:06.041353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:06.041318 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:06.477119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:06.476965 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6"] Oct 31 00:11:06.479879 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:11:06.479854 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2884d99_17e0_486e_ba7e_fec1605f05f2.slice/crio-fbf988ff776556f0ee4dc326b62adcad958558852b1dbe367f05902101441424 WatchSource:0}: Error finding container fbf988ff776556f0ee4dc326b62adcad958558852b1dbe367f05902101441424: Status 404 returned error can't find the container with id fbf988ff776556f0ee4dc326b62adcad958558852b1dbe367f05902101441424 Oct 31 00:11:07.001062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:07.001023 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" event={"ID":"c2884d99-17e0-486e-ba7e-fec1605f05f2","Type":"ContainerStarted","Data":"fbf988ff776556f0ee4dc326b62adcad958558852b1dbe367f05902101441424"} Oct 31 00:11:09.013195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:09.013142 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" event={"ID":"c2884d99-17e0-486e-ba7e-fec1605f05f2","Type":"ContainerStarted","Data":"ddadda8b0451467d7287f3c4b1e2408845370ad95122370ba775ae8926a97e84"} Oct 31 00:11:09.013612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:09.013312 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:09.181678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:09.181615 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" podStartSLOduration=2.175633381 podStartE2EDuration="4.181595906s" totalImagesPullingTime="2.005962525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:09.180800963 +0000 UTC m=+449.274091150" watchObservedRunningTime="2026-10-31 00:11:09.181595906 +0000 UTC m=+449.274886095" Oct 31 00:11:09.604218 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:09.604161 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:09.800578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:09.800547 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:10.415094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:10.415056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:10.816948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:10.816910 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:10.968094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:10.968055 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:13.813076 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:13.813045 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:14.003156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:14.003117 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:16.270851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:16.270809 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:16.462027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:16.461996 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:23.220103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:23.220024 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:23.364575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:23.364539 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:23.621127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:23.621091 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:23.770946 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:23.770912 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:23.913020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:23.912941 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:26.530724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:26.530690 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:26.681763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:26.681729 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:27.475048 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:11:27.475011 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:11:30.022568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:30.022532 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nmcf6" Oct 31 00:11:30.472455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:30.472376 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:30.622424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:30.622388 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:30.767390 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:30.767356 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:30.917606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:30.917573 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:31.171461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.171378 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f54bdd8cd-ncrz5"] Oct 31 00:11:31.175821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.175796 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.228414 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.228385 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 00:11:31.258100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.258066 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-metrics-certs\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.258291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.258153 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-webhook-cert\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.258347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.258311 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hw26\" (UniqueName: \"kubernetes.io/projected/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-kube-api-access-5hw26\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.277327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.277301 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 00:11:31.277327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.277300 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zq9pr\"" Oct 31 00:11:31.277327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.277306 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 00:11:31.278681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.278664 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f54bdd8cd-ncrz5"] Oct 31 00:11:31.359317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.359280 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hw26\" (UniqueName: \"kubernetes.io/projected/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-kube-api-access-5hw26\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.359518 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.359333 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-metrics-certs\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.359518 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.359389 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-webhook-cert\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.361888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.361865 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-webhook-cert\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.362016 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.361916 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-metrics-certs\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.420586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.420552 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hw26\" (UniqueName: \"kubernetes.io/projected/72141891-b5bf-4a1f-9fbe-aebe6e07bcc0-kube-api-access-5hw26\") pod \"maas-controller-5f54bdd8cd-ncrz5\" (UID: \"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0\") " pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.495636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.495549 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:31.871250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:31.871222 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f54bdd8cd-ncrz5"] Oct 31 00:11:31.873805 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:11:31.873782 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72141891_b5bf_4a1f_9fbe_aebe6e07bcc0.slice/crio-f07f7b0d287d769047d29eedc9fd538ac1289f656fe5fc86c9a7daa2aad0ff2d WatchSource:0}: Error finding container f07f7b0d287d769047d29eedc9fd538ac1289f656fe5fc86c9a7daa2aad0ff2d: Status 404 returned error can't find the container with id f07f7b0d287d769047d29eedc9fd538ac1289f656fe5fc86c9a7daa2aad0ff2d Oct 31 00:11:32.125120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:32.125039 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" event={"ID":"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0","Type":"ContainerStarted","Data":"f07f7b0d287d769047d29eedc9fd538ac1289f656fe5fc86c9a7daa2aad0ff2d"} Oct 31 00:11:32.238056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:32.238009 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:32.429299 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:32.429166 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:35.236479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:35.236443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:35.427421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:35.427383 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:37.707131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:37.707087 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:37.866102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:37.866070 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:40.463383 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:40.463287 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:43.420128 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:43.420099 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 00:11:43.741727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:43.741645 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:43.895476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:43.895443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:44.184580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:44.184540 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" event={"ID":"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0","Type":"ContainerStarted","Data":"8d45660ef97b05a5b20a47c4a5e939ee9842403cc1ce00199bd5c23e625e2bf8"} Oct 31 00:11:44.184770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:44.184646 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:44.301711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:44.301642 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" podStartSLOduration=1.808317684 podStartE2EDuration="13.30162642s" totalImagesPullingTime="11.493308736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:44.301287915 +0000 UTC m=+484.394578114" watchObservedRunningTime="2026-10-31 00:11:44.30162642 +0000 UTC m=+484.394916608" Oct 31 00:11:46.888328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:46.888289 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:47.124777 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:47.124737 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:55.191584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:11:55.191549 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:11:57.475890 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:11:57.475850 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:12:11.816107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:11.816064 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.010592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.010556 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.156662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.156577 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.311638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.311607 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.457244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.457134 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.604950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.604915 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.759995 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.759963 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:12.788519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.788489 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d"] Oct 31 00:12:12.794686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.794666 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:12.850096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.850056 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t4n2d\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:12.850660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.850245 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxjj\" (UniqueName: \"kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj\") pod \"llmisvc-controller-manager-869d5dcc5d-t4n2d\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:12.851448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.851428 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d"] Oct 31 00:12:12.899687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.899659 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 00:12:12.899851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.899716 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v6b99\"" Oct 31 00:12:12.950918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.950887 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t4n2d\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:12.951091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.950973 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxjj\" (UniqueName: \"kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj\") pod \"llmisvc-controller-manager-869d5dcc5d-t4n2d\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:12.953585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:12.953556 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t4n2d\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:13.012692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.012616 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5s5vk"] Oct 31 00:12:13.012851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.012808 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxjj\" (UniqueName: \"kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj\") pod \"llmisvc-controller-manager-869d5dcc5d-t4n2d\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:13.015088 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.015067 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.051800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.051760 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de47dd60-f7d9-4034-93a5-128c5fa025a7-metrics-certs\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.051980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.051891 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jn5c\" (UniqueName: \"kubernetes.io/projected/de47dd60-f7d9-4034-93a5-128c5fa025a7-kube-api-access-6jn5c\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.052053 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.051986 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de47dd60-f7d9-4034-93a5-128c5fa025a7-cert\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.075710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.075684 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5s5vk"] Oct 31 00:12:13.084592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.084566 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 00:12:13.084708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.084666 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 00:12:13.084708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.084695 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4vpqb\"" Oct 31 00:12:13.106357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.106324 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:13.152940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.152903 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de47dd60-f7d9-4034-93a5-128c5fa025a7-cert\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.153224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.153055 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de47dd60-f7d9-4034-93a5-128c5fa025a7-metrics-certs\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.153224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.153099 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jn5c\" (UniqueName: \"kubernetes.io/projected/de47dd60-f7d9-4034-93a5-128c5fa025a7-kube-api-access-6jn5c\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.157588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.157477 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de47dd60-f7d9-4034-93a5-128c5fa025a7-cert\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.157874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.157614 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de47dd60-f7d9-4034-93a5-128c5fa025a7-metrics-certs\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.214495 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.214465 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jn5c\" (UniqueName: \"kubernetes.io/projected/de47dd60-f7d9-4034-93a5-128c5fa025a7-kube-api-access-6jn5c\") pod \"odh-model-controller-769875db56-5s5vk\" (UID: \"de47dd60-f7d9-4034-93a5-128c5fa025a7\") " pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.330845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.330767 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:13.477961 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.477933 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d"] Oct 31 00:12:13.480477 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:12:13.480444 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc957f23a_38e6_474a_b01d_7f529a2ea881.slice/crio-43b9dc77d7530382bb4be42f4e061b54c6ae5c1d1f6a30d4483da5d5f0d26608 WatchSource:0}: Error finding container 43b9dc77d7530382bb4be42f4e061b54c6ae5c1d1f6a30d4483da5d5f0d26608: Status 404 returned error can't find the container with id 43b9dc77d7530382bb4be42f4e061b54c6ae5c1d1f6a30d4483da5d5f0d26608 Oct 31 00:12:13.709931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:13.709561 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5s5vk"] Oct 31 00:12:13.712417 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:12:13.712394 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde47dd60_f7d9_4034_93a5_128c5fa025a7.slice/crio-e9cb757b4ae0755632dbe729c3e1b8b57db02b6ad32cd09b895d1c7c07d996d2 WatchSource:0}: Error finding container e9cb757b4ae0755632dbe729c3e1b8b57db02b6ad32cd09b895d1c7c07d996d2: Status 404 returned error can't find the container with id e9cb757b4ae0755632dbe729c3e1b8b57db02b6ad32cd09b895d1c7c07d996d2 Oct 31 00:12:14.046963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:14.046930 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:14.324003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:14.323803 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5s5vk" event={"ID":"de47dd60-f7d9-4034-93a5-128c5fa025a7","Type":"ContainerStarted","Data":"e9cb757b4ae0755632dbe729c3e1b8b57db02b6ad32cd09b895d1c7c07d996d2"} Oct 31 00:12:14.325090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:14.325059 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" event={"ID":"c957f23a-38e6-474a-b01d-7f529a2ea881","Type":"ContainerStarted","Data":"43b9dc77d7530382bb4be42f4e061b54c6ae5c1d1f6a30d4483da5d5f0d26608"} Oct 31 00:12:15.532379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:15.532066 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:15.694974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:15.694856 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:17.345925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.345879 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5s5vk" event={"ID":"de47dd60-f7d9-4034-93a5-128c5fa025a7","Type":"ContainerStarted","Data":"5cf72c4fa6b43b082393d6b0c35e4a73de3e899092639cf6a917486bb8f7e54d"} Oct 31 00:12:17.346412 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.346018 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:17.347919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.347894 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" event={"ID":"c957f23a-38e6-474a-b01d-7f529a2ea881","Type":"ContainerStarted","Data":"789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6"} Oct 31 00:12:17.348047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.348025 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:17.461818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.461755 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5s5vk" podStartSLOduration=2.358153636 podStartE2EDuration="5.461735958s" totalImagesPullingTime="3.103582322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:17.460862204 +0000 UTC m=+517.554152409" watchObservedRunningTime="2026-10-31 00:12:17.461735958 +0000 UTC m=+517.555026146" Oct 31 00:12:17.577433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.577378 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" podStartSLOduration=2.773798626 podStartE2EDuration="5.577362221s" totalImagesPullingTime="2.803563595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:17.577059233 +0000 UTC m=+517.670349454" watchObservedRunningTime="2026-10-31 00:12:17.577362221 +0000 UTC m=+517.670652409" Oct 31 00:12:17.582976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.582945 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:17.729862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.729755 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:17.882577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:17.882542 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:18.066119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:18.066087 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.158499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:19.158461 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.354105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:19.354071 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.500578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:19.500499 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.648189 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:19.648139 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.798768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:19.798740 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:19.999616 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:19.999583 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:20.200115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:20.200019 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:20.346696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:20.346658 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:23.526139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:23.526093 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:12:27.475532 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:12:27.475429 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:12:28.354719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:28.354689 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5s5vk" Oct 31 00:12:31.707981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:31.707921 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:32.040489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.040454 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-khgsd"] Oct 31 00:12:32.043517 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.043501 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:32.097932 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.097889 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7p7cn\"" Oct 31 00:12:32.133586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.133547 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcpz\" (UniqueName: \"kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz\") pod \"authorino-6f6944bcdd-khgsd\" (UID: \"c58e4bb1-4e9e-4dce-956b-6d429d514e1f\") " pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:32.148653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.148626 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-khgsd"] Oct 31 00:12:32.234340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.234306 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcpz\" (UniqueName: \"kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz\") pod \"authorino-6f6944bcdd-khgsd\" (UID: \"c58e4bb1-4e9e-4dce-956b-6d429d514e1f\") " pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:32.295708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.295641 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcpz\" (UniqueName: \"kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz\") pod \"authorino-6f6944bcdd-khgsd\" (UID: \"c58e4bb1-4e9e-4dce-956b-6d429d514e1f\") " pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:32.353913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.353875 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:32.415428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.415398 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-khgsd"] Oct 31 00:12:32.443208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.443159 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-647848b5d-9r5cm"] Oct 31 00:12:32.446116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.446098 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:32.504155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.504120 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-647848b5d-9r5cm"] Oct 31 00:12:32.537340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.537310 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5r22\" (UniqueName: \"kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22\") pod \"authorino-647848b5d-9r5cm\" (UID: \"d4911c42-75c5-41df-b0b0-d68a6e874cfe\") " pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:32.638226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.638065 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5r22\" (UniqueName: \"kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22\") pod \"authorino-647848b5d-9r5cm\" (UID: \"d4911c42-75c5-41df-b0b0-d68a6e874cfe\") " pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:32.700059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.700022 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5r22\" (UniqueName: \"kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22\") pod \"authorino-647848b5d-9r5cm\" (UID: \"d4911c42-75c5-41df-b0b0-d68a6e874cfe\") " pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:32.725820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.725794 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-khgsd"] Oct 31 00:12:32.728120 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:12:32.728093 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc58e4bb1_4e9e_4dce_956b_6d429d514e1f.slice/crio-2be777e0dd2a5122c45dd6ef1c2ab8df30b4eab4b715fe61fa26aaa27564e1ee WatchSource:0}: Error finding container 2be777e0dd2a5122c45dd6ef1c2ab8df30b4eab4b715fe61fa26aaa27564e1ee: Status 404 returned error can't find the container with id 2be777e0dd2a5122c45dd6ef1c2ab8df30b4eab4b715fe61fa26aaa27564e1ee Oct 31 00:12:32.756926 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.756897 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:32.849261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.849230 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-647848b5d-9r5cm"] Oct 31 00:12:32.875801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.875766 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b49fc69d8-xgzhv"] Oct 31 00:12:32.878417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.878397 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:32.931323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.931249 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 00:12:32.937398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.937371 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b49fc69d8-xgzhv"] Oct 31 00:12:32.941188 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.941145 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert\") pod \"authorino-6b49fc69d8-xgzhv\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:32.941329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:32.941306 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvbl\" (UniqueName: \"kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl\") pod \"authorino-6b49fc69d8-xgzhv\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:33.042568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.042532 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert\") pod \"authorino-6b49fc69d8-xgzhv\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:33.042737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.042585 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvbl\" (UniqueName: \"kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl\") pod \"authorino-6b49fc69d8-xgzhv\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:33.045089 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.045061 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert\") pod \"authorino-6b49fc69d8-xgzhv\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:33.103344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.103310 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvbl\" (UniqueName: \"kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl\") pod \"authorino-6b49fc69d8-xgzhv\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:33.108376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.108333 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:33.138345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.138317 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-647848b5d-9r5cm"] Oct 31 00:12:33.189238 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.189146 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:12:33.423744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.423702 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" event={"ID":"c58e4bb1-4e9e-4dce-956b-6d429d514e1f","Type":"ContainerStarted","Data":"2be777e0dd2a5122c45dd6ef1c2ab8df30b4eab4b715fe61fa26aaa27564e1ee"} Oct 31 00:12:33.424898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.424868 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647848b5d-9r5cm" event={"ID":"d4911c42-75c5-41df-b0b0-d68a6e874cfe","Type":"ContainerStarted","Data":"fbb8731cab39641232d46d64402b634675f84af6c59d59179b18082b6d766db8"} Oct 31 00:12:33.564419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:33.563896 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b49fc69d8-xgzhv"] Oct 31 00:12:33.566087 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:12:33.566059 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aa39741_74b9_43a2_a551_d8ff5fb0ab73.slice/crio-087553ef1fa883452c569f9b60716e7c2e9ffc88c8e41396ad5331b26d5b8554 WatchSource:0}: Error finding container 087553ef1fa883452c569f9b60716e7c2e9ffc88c8e41396ad5331b26d5b8554: Status 404 returned error can't find the container with id 087553ef1fa883452c569f9b60716e7c2e9ffc88c8e41396ad5331b26d5b8554 Oct 31 00:12:34.430637 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:34.430597 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" event={"ID":"6aa39741-74b9-43a2-a551-d8ff5fb0ab73","Type":"ContainerStarted","Data":"087553ef1fa883452c569f9b60716e7c2e9ffc88c8e41396ad5331b26d5b8554"} Oct 31 00:12:35.437533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.437422 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" event={"ID":"c58e4bb1-4e9e-4dce-956b-6d429d514e1f","Type":"ContainerStarted","Data":"3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b"} Oct 31 00:12:35.437533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.437442 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" podUID="c58e4bb1-4e9e-4dce-956b-6d429d514e1f" containerName="authorino" containerID="cri-o://3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b" gracePeriod=30 Oct 31 00:12:35.439305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.439257 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647848b5d-9r5cm" event={"ID":"d4911c42-75c5-41df-b0b0-d68a6e874cfe","Type":"ContainerStarted","Data":"72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df"} Oct 31 00:12:35.439442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.439293 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-647848b5d-9r5cm" podUID="d4911c42-75c5-41df-b0b0-d68a6e874cfe" containerName="authorino" containerID="cri-o://72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df" gracePeriod=30 Oct 31 00:12:35.441508 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.441476 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" event={"ID":"6aa39741-74b9-43a2-a551-d8ff5fb0ab73","Type":"ContainerStarted","Data":"4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb"} Oct 31 00:12:35.556391 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.555696 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" podStartSLOduration=2.669938128 podStartE2EDuration="4.555680965s" totalImagesPullingTime="1.885742837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:35.555390687 +0000 UTC m=+535.648680879" watchObservedRunningTime="2026-10-31 00:12:35.555680965 +0000 UTC m=+535.648971153" Oct 31 00:12:35.672605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.672542 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" podStartSLOduration=2.174010658 podStartE2EDuration="3.672522596s" totalImagesPullingTime="1.498511938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:35.67034934 +0000 UTC m=+535.763639534" watchObservedRunningTime="2026-10-31 00:12:35.672522596 +0000 UTC m=+535.765812784" Oct 31 00:12:35.741301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.741145 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:35.744956 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.744936 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:35.792104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.792047 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-647848b5d-9r5cm" podStartSLOduration=2.300718927 podStartE2EDuration="3.79203297s" totalImagesPullingTime="1.491314043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:35.791609678 +0000 UTC m=+535.884899878" watchObservedRunningTime="2026-10-31 00:12:35.79203297 +0000 UTC m=+535.885323183" Oct 31 00:12:35.874434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.874394 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz\" (UniqueName: \"kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz\") pod \"c58e4bb1-4e9e-4dce-956b-6d429d514e1f\" (UID: \"c58e4bb1-4e9e-4dce-956b-6d429d514e1f\") " Oct 31 00:12:35.874639 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.874505 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22\" (UniqueName: \"kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22\") pod \"d4911c42-75c5-41df-b0b0-d68a6e874cfe\" (UID: \"d4911c42-75c5-41df-b0b0-d68a6e874cfe\") " Oct 31 00:12:35.876809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.876774 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz" pod "c58e4bb1-4e9e-4dce-956b-6d429d514e1f" (UID: "c58e4bb1-4e9e-4dce-956b-6d429d514e1f"). InnerVolumeSpecName "kube-api-access-mtcpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:35.876937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.876813 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22" pod "d4911c42-75c5-41df-b0b0-d68a6e874cfe" (UID: "d4911c42-75c5-41df-b0b0-d68a6e874cfe"). InnerVolumeSpecName "kube-api-access-m5r22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:35.975648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.975557 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcpz\" (UniqueName: \"kubernetes.io/projected/c58e4bb1-4e9e-4dce-956b-6d429d514e1f-kube-api-access-mtcpz\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:12:35.975648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:35.975593 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5r22\" (UniqueName: \"kubernetes.io/projected/d4911c42-75c5-41df-b0b0-d68a6e874cfe-kube-api-access-m5r22\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:12:36.447572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.447533 5943 generic.go:356] "Generic (PLEG): container finished" podID="c58e4bb1-4e9e-4dce-956b-6d429d514e1f" containerID="3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b" exitCode=0 Oct 31 00:12:36.448070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.447587 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" Oct 31 00:12:36.448070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.447628 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" event={"ID":"c58e4bb1-4e9e-4dce-956b-6d429d514e1f","Type":"ContainerDied","Data":"3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b"} Oct 31 00:12:36.448070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.447671 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-khgsd" event={"ID":"c58e4bb1-4e9e-4dce-956b-6d429d514e1f","Type":"ContainerDied","Data":"2be777e0dd2a5122c45dd6ef1c2ab8df30b4eab4b715fe61fa26aaa27564e1ee"} Oct 31 00:12:36.448070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.447689 5943 scope.go:122] "RemoveContainer" containerID="3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b" Oct 31 00:12:36.450937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.450902 5943 generic.go:356] "Generic (PLEG): container finished" podID="d4911c42-75c5-41df-b0b0-d68a6e874cfe" containerID="72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df" exitCode=0 Oct 31 00:12:36.450937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.450940 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647848b5d-9r5cm" Oct 31 00:12:36.451398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.450975 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647848b5d-9r5cm" event={"ID":"d4911c42-75c5-41df-b0b0-d68a6e874cfe","Type":"ContainerDied","Data":"72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df"} Oct 31 00:12:36.451398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.451012 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647848b5d-9r5cm" event={"ID":"d4911c42-75c5-41df-b0b0-d68a6e874cfe","Type":"ContainerDied","Data":"fbb8731cab39641232d46d64402b634675f84af6c59d59179b18082b6d766db8"} Oct 31 00:12:36.458350 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.458331 5943 scope.go:122] "RemoveContainer" containerID="3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b" Oct 31 00:12:36.458605 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:12:36.458589 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b\": container with ID starting with 3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b not found: ID does not exist" containerID="3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b" Oct 31 00:12:36.458645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.458616 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b"} err="failed to get container status \"3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b\": rpc error: code = NotFound desc = could not find container \"3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b\": container with ID starting with 3d3f43040d6be43efb264732c690359431fed51b712dd01d580d5819b15a0c3b not found: ID does not exist" Oct 31 00:12:36.458645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.458633 5943 scope.go:122] "RemoveContainer" containerID="72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df" Oct 31 00:12:36.466304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.466286 5943 scope.go:122] "RemoveContainer" containerID="72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df" Oct 31 00:12:36.466575 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:12:36.466556 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df\": container with ID starting with 72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df not found: ID does not exist" containerID="72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df" Oct 31 00:12:36.466632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.466584 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df"} err="failed to get container status \"72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df\": rpc error: code = NotFound desc = could not find container \"72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df\": container with ID starting with 72d86c38bd8059933019bbd48db00696762db6ae856f9bb15ed9b147750ea4df not found: ID does not exist" Oct 31 00:12:36.609556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.609522 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-khgsd"] Oct 31 00:12:36.658859 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.658822 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-khgsd"] Oct 31 00:12:36.767553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.767522 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-647848b5d-9r5cm"] Oct 31 00:12:36.773640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:36.773612 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-647848b5d-9r5cm"] Oct 31 00:12:38.039787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:38.039755 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58e4bb1-4e9e-4dce-956b-6d429d514e1f" path="/var/lib/kubelet/pods/c58e4bb1-4e9e-4dce-956b-6d429d514e1f/volumes" Oct 31 00:12:38.040204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:38.040111 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4911c42-75c5-41df-b0b0-d68a6e874cfe" path="/var/lib/kubelet/pods/d4911c42-75c5-41df-b0b0-d68a6e874cfe/volumes" Oct 31 00:12:40.818303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:40.818266 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:46.738464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:46.738428 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:48.354078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:12:48.354046 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:12:57.475376 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:12:57.475290 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:13:23.634149 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:23.634091 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:23.825691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:23.825641 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:27.474952 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:13:27.474916 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:13:37.251605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.251571 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6"] Oct 31 00:13:37.251984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.251959 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c58e4bb1-4e9e-4dce-956b-6d429d514e1f" containerName="authorino" Oct 31 00:13:37.251984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.251970 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58e4bb1-4e9e-4dce-956b-6d429d514e1f" containerName="authorino" Oct 31 00:13:37.252071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.251989 5943 cpu_manager.go:414] "containerMap: removing container" podUID="d4911c42-75c5-41df-b0b0-d68a6e874cfe" containerName="authorino" Oct 31 00:13:37.252071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.251995 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4911c42-75c5-41df-b0b0-d68a6e874cfe" containerName="authorino" Oct 31 00:13:37.252071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.252069 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58e4bb1-4e9e-4dce-956b-6d429d514e1f" containerName="authorino" Oct 31 00:13:37.252225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.252078 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4911c42-75c5-41df-b0b0-d68a6e874cfe" containerName="authorino" Oct 31 00:13:37.254079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.254064 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.324192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.324155 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6"] Oct 31 00:13:37.349782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.349744 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dn7\" (UniqueName: \"kubernetes.io/projected/b5e2db63-a291-4143-8832-9d64a03e8484-kube-api-access-k7dn7\") pod \"llmisvc-controller-manager-6466c554d8-8ldm6\" (UID: \"b5e2db63-a291-4143-8832-9d64a03e8484\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.349947 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.349883 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5e2db63-a291-4143-8832-9d64a03e8484-cert\") pod \"llmisvc-controller-manager-6466c554d8-8ldm6\" (UID: \"b5e2db63-a291-4143-8832-9d64a03e8484\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.451262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.451219 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5e2db63-a291-4143-8832-9d64a03e8484-cert\") pod \"llmisvc-controller-manager-6466c554d8-8ldm6\" (UID: \"b5e2db63-a291-4143-8832-9d64a03e8484\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.451467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.451305 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dn7\" (UniqueName: \"kubernetes.io/projected/b5e2db63-a291-4143-8832-9d64a03e8484-kube-api-access-k7dn7\") pod \"llmisvc-controller-manager-6466c554d8-8ldm6\" (UID: \"b5e2db63-a291-4143-8832-9d64a03e8484\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.453903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.453877 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5e2db63-a291-4143-8832-9d64a03e8484-cert\") pod \"llmisvc-controller-manager-6466c554d8-8ldm6\" (UID: \"b5e2db63-a291-4143-8832-9d64a03e8484\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.512562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.512527 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dn7\" (UniqueName: \"kubernetes.io/projected/b5e2db63-a291-4143-8832-9d64a03e8484-kube-api-access-k7dn7\") pod \"llmisvc-controller-manager-6466c554d8-8ldm6\" (UID: \"b5e2db63-a291-4143-8832-9d64a03e8484\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.565873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.565831 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:37.935302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:37.935233 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6"] Oct 31 00:13:37.938045 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:13:37.938003 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e2db63_a291_4143_8832_9d64a03e8484.slice/crio-2930a9e16203f77a4c2f235ee495fc107f4a58202c674d7ffbaaff9022a9be17 WatchSource:0}: Error finding container 2930a9e16203f77a4c2f235ee495fc107f4a58202c674d7ffbaaff9022a9be17: Status 404 returned error can't find the container with id 2930a9e16203f77a4c2f235ee495fc107f4a58202c674d7ffbaaff9022a9be17 Oct 31 00:13:38.750525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:38.750492 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" event={"ID":"b5e2db63-a291-4143-8832-9d64a03e8484","Type":"ContainerStarted","Data":"2930a9e16203f77a4c2f235ee495fc107f4a58202c674d7ffbaaff9022a9be17"} Oct 31 00:13:39.757505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:39.757466 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" event={"ID":"b5e2db63-a291-4143-8832-9d64a03e8484","Type":"ContainerStarted","Data":"319e86e7f4dff9ba01da7bd483bef0a928e48307ca3a0d15c4ccffdf4fca8542"} Oct 31 00:13:39.757993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:39.757580 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:13:39.873126 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:39.873062 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" podStartSLOduration=2.108843938 podStartE2EDuration="2.873044002s" totalImagesPullingTime="764.200064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:39.87249508 +0000 UTC m=+599.965785313" watchObservedRunningTime="2026-10-31 00:13:39.873044002 +0000 UTC m=+599.966334189" Oct 31 00:13:41.171513 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:13:41.171476 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:57.475159 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:13:57.475120 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:14:06.703671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:06.703614 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:10.765116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:10.765085 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8ldm6" Oct 31 00:14:10.933244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:10.933209 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d"] Oct 31 00:14:10.933521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:10.933494 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" podUID="c957f23a-38e6-474a-b01d-7f529a2ea881" containerName="manager" containerID="cri-o://789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6" gracePeriod=30 Oct 31 00:14:11.179140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.179116 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:14:11.272076 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.272041 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert\" (UniqueName: \"kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert\") pod \"c957f23a-38e6-474a-b01d-7f529a2ea881\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " Oct 31 00:14:11.272309 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.272106 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj\" (UniqueName: \"kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj\") pod \"c957f23a-38e6-474a-b01d-7f529a2ea881\" (UID: \"c957f23a-38e6-474a-b01d-7f529a2ea881\") " Oct 31 00:14:11.274510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.274482 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj" pod "c957f23a-38e6-474a-b01d-7f529a2ea881" (UID: "c957f23a-38e6-474a-b01d-7f529a2ea881"). InnerVolumeSpecName "kube-api-access-2hxjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:11.274626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.274522 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert" pod "c957f23a-38e6-474a-b01d-7f529a2ea881" (UID: "c957f23a-38e6-474a-b01d-7f529a2ea881"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:11.373576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.373486 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxjj\" (UniqueName: \"kubernetes.io/projected/c957f23a-38e6-474a-b01d-7f529a2ea881-kube-api-access-2hxjj\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:14:11.373576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.373520 5943 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c957f23a-38e6-474a-b01d-7f529a2ea881-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:14:11.914061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.914026 5943 generic.go:356] "Generic (PLEG): container finished" podID="c957f23a-38e6-474a-b01d-7f529a2ea881" containerID="789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6" exitCode=0 Oct 31 00:14:11.914593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.914074 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" Oct 31 00:14:11.914593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.914119 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" event={"ID":"c957f23a-38e6-474a-b01d-7f529a2ea881","Type":"ContainerDied","Data":"789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6"} Oct 31 00:14:11.914593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.914201 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d" event={"ID":"c957f23a-38e6-474a-b01d-7f529a2ea881","Type":"ContainerDied","Data":"43b9dc77d7530382bb4be42f4e061b54c6ae5c1d1f6a30d4483da5d5f0d26608"} Oct 31 00:14:11.914593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.914241 5943 scope.go:122] "RemoveContainer" containerID="789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6" Oct 31 00:14:11.923244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.923228 5943 scope.go:122] "RemoveContainer" containerID="789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6" Oct 31 00:14:11.923538 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:14:11.923519 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6\": container with ID starting with 789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6 not found: ID does not exist" containerID="789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6" Oct 31 00:14:11.923590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:11.923548 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6"} err="failed to get container status \"789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6\": rpc error: code = NotFound desc = could not find container \"789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6\": container with ID starting with 789694aed63922c4e49bd08ad70378b89831badf5bb03abbd3f32444842164f6 not found: ID does not exist" Oct 31 00:14:12.076920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:12.076884 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d"] Oct 31 00:14:12.086448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:12.086421 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t4n2d"] Oct 31 00:14:12.098716 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:12.098677 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:12.599709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:12.599673 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:13.927514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:13.927476 5943 generic.go:356] "Generic (PLEG): container finished" podID="72141891-b5bf-4a1f-9fbe-aebe6e07bcc0" containerID="8d45660ef97b05a5b20a47c4a5e939ee9842403cc1ce00199bd5c23e625e2bf8" exitCode=1 Oct 31 00:14:13.927914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:13.927548 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" event={"ID":"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0","Type":"ContainerDied","Data":"8d45660ef97b05a5b20a47c4a5e939ee9842403cc1ce00199bd5c23e625e2bf8"} Oct 31 00:14:13.927961 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:13.927913 5943 scope.go:122] "RemoveContainer" containerID="8d45660ef97b05a5b20a47c4a5e939ee9842403cc1ce00199bd5c23e625e2bf8" Oct 31 00:14:14.039343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:14.039312 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c957f23a-38e6-474a-b01d-7f529a2ea881" path="/var/lib/kubelet/pods/c957f23a-38e6-474a-b01d-7f529a2ea881/volumes" Oct 31 00:14:14.934690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:14.934654 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" event={"ID":"72141891-b5bf-4a1f-9fbe-aebe6e07bcc0","Type":"ContainerStarted","Data":"4424c5b9cf7db132cc3c8dfac0c978b4a257b0ad83738ab8e96914ea6661dffb"} Oct 31 00:14:14.935074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:14.934888 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:14:19.285973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:19.285877 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.433556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:19.433521 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:25.942326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:25.942295 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f54bdd8cd-ncrz5" Oct 31 00:14:27.475592 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:14:27.475550 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:14:41.520895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:41.520855 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:42.588169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:42.588135 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:43.156432 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:43.156394 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:44.587241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:44.587209 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:44.802173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:44.802135 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:51.252638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:51.252606 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:54.680992 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:54.680955 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:14:57.474848 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:14:57.474806 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:14:57.923538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:14:57.923504 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:01.856524 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:01.856486 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:05.281798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:05.281767 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:05.721710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:05.721630 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:15:11.029599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:11.029558 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:14.590839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:14.590801 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:15:16.216919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:16.216882 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:19.850442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:19.850403 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.623386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:23.623354 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:27.063674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:27.063644 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:15:27.491377 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:15:27.491285 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:15:29.609467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:29.609434 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:41.472216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.472162 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z"] Oct 31 00:15:41.472601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.472569 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c957f23a-38e6-474a-b01d-7f529a2ea881" containerName="manager" Oct 31 00:15:41.472601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.472580 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c957f23a-38e6-474a-b01d-7f529a2ea881" containerName="manager" Oct 31 00:15:41.472677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.472661 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c957f23a-38e6-474a-b01d-7f529a2ea881" containerName="manager" Oct 31 00:15:41.475922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.475901 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.528345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.528315 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r4dfh\"" Oct 31 00:15:41.541776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.541747 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.541776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.541776 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.541954 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.541800 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.541954 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.541910 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.542022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.541955 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.542022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.542010 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.542087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.542026 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.542087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.542049 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw5s\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.542162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.542144 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.589851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.589819 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z"] Oct 31 00:15:41.643510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643476 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643540 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643570 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643599 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643636 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643670 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643712 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643740 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643771 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw5s\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.643982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.643962 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.644173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.644028 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.644173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.644087 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.644292 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.644272 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.644455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.644432 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.646127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.646101 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.646271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.646208 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.715244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.715213 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.715913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.715884 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw5s\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s\") pod \"maas-default-gateway-openshift-default-58c6585447-7v26z\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.788825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.788797 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:41.872466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:41.872433 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:42.219662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.219591 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z"] Oct 31 00:15:42.222548 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:15:42.222516 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84585842_bbef_4602_bf0d_b6e9cd15d756.slice/crio-62ba0c0116d9f46d736144f68714ee10b7b7a9d148789034ec661f6fc4474f47 WatchSource:0}: Error finding container 62ba0c0116d9f46d736144f68714ee10b7b7a9d148789034ec661f6fc4474f47: Status 404 returned error can't find the container with id 62ba0c0116d9f46d736144f68714ee10b7b7a9d148789034ec661f6fc4474f47 Oct 31 00:15:42.224805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.224779 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:15:42.224873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.224843 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:15:42.224909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.224876 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:15:42.350306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.350266 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" event={"ID":"84585842-bbef-4602-bf0d-b6e9cd15d756","Type":"ContainerStarted","Data":"3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9"} Oct 31 00:15:42.350469 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.350324 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" event={"ID":"84585842-bbef-4602-bf0d-b6e9cd15d756","Type":"ContainerStarted","Data":"62ba0c0116d9f46d736144f68714ee10b7b7a9d148789034ec661f6fc4474f47"} Oct 31 00:15:42.471344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.471240 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podStartSLOduration=1.471226176 podStartE2EDuration="1.471226176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:42.47068643 +0000 UTC m=+722.563976638" watchObservedRunningTime="2026-10-31 00:15:42.471226176 +0000 UTC m=+722.564516364" Oct 31 00:15:42.789562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:42.789532 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:43.791454 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:43.791418 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7v26z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 00:15:43.791850 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:43.791477 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 00:15:44.790138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:44.790104 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7v26z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 00:15:44.790343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:44.790156 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 00:15:45.789483 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:45.789449 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7v26z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 00:15:45.789874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:45.789514 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 00:15:46.789236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:46.789195 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7v26z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 00:15:46.789431 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:46.789278 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 00:15:47.789871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:47.789838 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7v26z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 00:15:47.790307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:47.789889 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 00:15:48.501635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:48.501598 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:48.650609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:48.650570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:48.789933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:48.789889 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7v26z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 00:15:48.790479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:48.789955 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 00:15:50.078231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:50.078149 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:50.283599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:50.283555 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:50.473503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:50.473414 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:50.473853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:50.473814 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:50.474619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:50.474599 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:15:51.732681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:51.732649 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:52.021076 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:52.021047 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:53.130953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.130916 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b49fc69d8-xgzhv"] Oct 31 00:15:53.131376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.131149 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" podUID="6aa39741-74b9-43a2-a551-d8ff5fb0ab73" containerName="authorino" containerID="cri-o://4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb" gracePeriod=30 Oct 31 00:15:53.379046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.379020 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:15:53.405056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.404986 5943 generic.go:356] "Generic (PLEG): container finished" podID="6aa39741-74b9-43a2-a551-d8ff5fb0ab73" containerID="4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb" exitCode=0 Oct 31 00:15:53.405056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.405035 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" Oct 31 00:15:53.405234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.405059 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" event={"ID":"6aa39741-74b9-43a2-a551-d8ff5fb0ab73","Type":"ContainerDied","Data":"4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb"} Oct 31 00:15:53.405234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.405095 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b49fc69d8-xgzhv" event={"ID":"6aa39741-74b9-43a2-a551-d8ff5fb0ab73","Type":"ContainerDied","Data":"087553ef1fa883452c569f9b60716e7c2e9ffc88c8e41396ad5331b26d5b8554"} Oct 31 00:15:53.405234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.405117 5943 scope.go:122] "RemoveContainer" containerID="4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb" Oct 31 00:15:53.413773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.413753 5943 scope.go:122] "RemoveContainer" containerID="4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb" Oct 31 00:15:53.414057 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:15:53.414035 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb\": container with ID starting with 4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb not found: ID does not exist" containerID="4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb" Oct 31 00:15:53.414098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.414068 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb"} err="failed to get container status \"4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb\": rpc error: code = NotFound desc = could not find container \"4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb\": container with ID starting with 4983270d8b197b2a5a4ea2fdedf28c4b322e8468b500dbee4e5be04cdcfed6bb not found: ID does not exist" Oct 31 00:15:53.463451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.463421 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl\" (UniqueName: \"kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl\") pod \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " Oct 31 00:15:53.463583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.463480 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert\" (UniqueName: \"kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert\") pod \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\" (UID: \"6aa39741-74b9-43a2-a551-d8ff5fb0ab73\") " Oct 31 00:15:53.465678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.465653 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl" pod "6aa39741-74b9-43a2-a551-d8ff5fb0ab73" (UID: "6aa39741-74b9-43a2-a551-d8ff5fb0ab73"). InnerVolumeSpecName "kube-api-access-mqvbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:53.476429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.476401 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert" pod "6aa39741-74b9-43a2-a551-d8ff5fb0ab73" (UID: "6aa39741-74b9-43a2-a551-d8ff5fb0ab73"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:53.564354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.564320 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqvbl\" (UniqueName: \"kubernetes.io/projected/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-kube-api-access-mqvbl\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:15:53.564354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.564344 5943 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6aa39741-74b9-43a2-a551-d8ff5fb0ab73-tls-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:15:53.823876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.823842 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b49fc69d8-xgzhv"] Oct 31 00:15:53.828334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:53.828309 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b49fc69d8-xgzhv"] Oct 31 00:15:54.045870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:15:54.045834 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa39741-74b9-43a2-a551-d8ff5fb0ab73" path="/var/lib/kubelet/pods/6aa39741-74b9-43a2-a551-d8ff5fb0ab73/volumes" Oct 31 00:15:57.475400 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:15:57.475360 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:16:25.117313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:25.117269 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.309549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:25.309520 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.557287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:25.557242 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.823373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:25.823253 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.973563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:25.973514 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:26.124867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:26.124781 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:26.273513 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:26.273471 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:26.473565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:26.473481 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:26.748549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:26.748460 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:27.021649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:27.021617 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:27.206371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:27.206340 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:27.474690 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:16:27.474604 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:16:28.905735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:28.905704 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.060469 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:29.060434 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.665212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:29.665167 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.844382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:29.844343 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:30.025664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:30.025635 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:30.288899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:30.288789 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:30.477436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:30.477375 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:30.630325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:30.630232 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:31.156939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:31.156903 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:31.370074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:31.370041 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:32.632408 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:32.632376 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:32.788829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:32.788793 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:32.976775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:32.976689 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:33.637443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:33.637410 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:33.853745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:33.853698 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.010685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:34.010650 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:36.111091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:36.111056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:36.271699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:36.271656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:36.422416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:36.422303 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:36.572835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:36.572800 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:36.742020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:36.741918 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:38.575787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:38.575755 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:38.753693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:38.753660 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.947383 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:39.947349 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:40.242885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:40.242770 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.147913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.147856 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.328081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.328047 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.597701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.597663 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.702521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.702486 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf"] Oct 31 00:16:41.703156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.703130 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6aa39741-74b9-43a2-a551-d8ff5fb0ab73" containerName="authorino" Oct 31 00:16:41.703156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.703155 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa39741-74b9-43a2-a551-d8ff5fb0ab73" containerName="authorino" Oct 31 00:16:41.703374 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.703295 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa39741-74b9-43a2-a551-d8ff5fb0ab73" containerName="authorino" Oct 31 00:16:41.707976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.707952 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.787435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.787387 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e1bbe4b\"" Oct 31 00:16:41.787435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.787416 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e1bbe4b-dockercfg-ntjbq\"" Oct 31 00:16:41.799063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.799036 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.799244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.799099 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglhf\" (UniqueName: \"kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf\") pod \"payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.806907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.806877 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.810595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.810577 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf"] Oct 31 00:16:41.899739 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.899664 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglhf\" (UniqueName: \"kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf\") pod \"payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.899739 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.899732 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.900389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.900371 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.960580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.960543 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglhf\" (UniqueName: \"kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf\") pod \"payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:41.965852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:41.965822 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:42.021150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:42.021114 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:42.227478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:42.227378 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:42.393231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:42.393205 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf"] Oct 31 00:16:42.396071 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:16:42.396040 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fd454f_fef4_4b8e_8072_5417811ad308.slice/crio-c5ff1e6bf6300a836a6023cdc5a5f08d755c832fa27f9329928e445d0aca1996 WatchSource:0}: Error finding container c5ff1e6bf6300a836a6023cdc5a5f08d755c832fa27f9329928e445d0aca1996: Status 404 returned error can't find the container with id c5ff1e6bf6300a836a6023cdc5a5f08d755c832fa27f9329928e445d0aca1996 Oct 31 00:16:42.398061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:42.398045 5943 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:16:42.639689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:42.639652 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" event={"ID":"b9fd454f-fef4-4b8e-8072-5417811ad308","Type":"ContainerStarted","Data":"c5ff1e6bf6300a836a6023cdc5a5f08d755c832fa27f9329928e445d0aca1996"} Oct 31 00:16:43.676512 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:43.676470 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:43.865849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:43.865817 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:45.661287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:45.661251 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" event={"ID":"b9fd454f-fef4-4b8e-8072-5417811ad308","Type":"ContainerStarted","Data":"d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6"} Oct 31 00:16:45.661672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:45.661406 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:45.674302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:45.674262 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:16:45.784847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:45.784787 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" podStartSLOduration=2.143910725 podStartE2EDuration="4.784773021s" totalImagesPullingTime="2.640862296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:45.783800076 +0000 UTC m=+785.877090266" watchObservedRunningTime="2026-10-31 00:16:45.784773021 +0000 UTC m=+785.878063207" Oct 31 00:16:46.400351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:46.400317 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:46.561563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:46.561512 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:46.704940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:46.704861 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:51.368230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:51.368164 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:53.057485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.057449 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:53.253720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.253690 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:53.476592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.476504 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl"] Oct 31 00:16:53.480154 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.480139 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.581606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.581562 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b83a178e-dockercfg-pdf7l\"" Oct 31 00:16:53.581606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.581562 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b83a178e\"" Oct 31 00:16:53.583305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.583285 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl"] Oct 31 00:16:53.615103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.615066 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.615103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.615100 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hvp\" (UniqueName: \"kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp\") pod \"payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.632663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.632625 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx"] Oct 31 00:16:53.636133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.636117 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.690413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.690355 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx"] Oct 31 00:16:53.716386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.716348 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.716586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.716388 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hvp\" (UniqueName: \"kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp\") pod \"payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.716586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.716480 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gg8n\" (UniqueName: \"kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n\") pod \"payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.716586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.716511 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume\") pod \"payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.717131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.717104 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.778715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.778688 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hvp\" (UniqueName: \"kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp\") pod \"payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.792664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.792640 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:53.818023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.817989 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume\") pod \"payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.818204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.818160 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gg8n\" (UniqueName: \"kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n\") pod \"payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.818682 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.818664 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume\") pod \"payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.879503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.879468 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gg8n\" (UniqueName: \"kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n\") pod \"payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:53.949768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:53.949729 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:54.164366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.164341 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl"] Oct 31 00:16:54.166575 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:16:54.166550 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf82392_9e81_4e0d_9568_c2d37cd622cc.slice/crio-0bb86ea21592f3193f35cb5a1a163f75736cdf32265cce62d0de913b6abc1e8d WatchSource:0}: Error finding container 0bb86ea21592f3193f35cb5a1a163f75736cdf32265cce62d0de913b6abc1e8d: Status 404 returned error can't find the container with id 0bb86ea21592f3193f35cb5a1a163f75736cdf32265cce62d0de913b6abc1e8d Oct 31 00:16:54.321802 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.321770 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx"] Oct 31 00:16:54.324961 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:16:54.324936 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3324d3b7_d44b_4905_bf8c_6346891a7189.slice/crio-69b694dcad3d857b645e90e9eb08783bf5e278fbbff248ef26d6ff971c5996e6 WatchSource:0}: Error finding container 69b694dcad3d857b645e90e9eb08783bf5e278fbbff248ef26d6ff971c5996e6: Status 404 returned error can't find the container with id 69b694dcad3d857b645e90e9eb08783bf5e278fbbff248ef26d6ff971c5996e6 Oct 31 00:16:54.384811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.384775 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:16:54.708347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.708315 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" event={"ID":"3cf82392-9e81-4e0d-9568-c2d37cd622cc","Type":"ContainerStarted","Data":"f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9"} Oct 31 00:16:54.708559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.708377 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" event={"ID":"3cf82392-9e81-4e0d-9568-c2d37cd622cc","Type":"ContainerStarted","Data":"0bb86ea21592f3193f35cb5a1a163f75736cdf32265cce62d0de913b6abc1e8d"} Oct 31 00:16:54.708559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.708420 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:16:54.710064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.710039 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" event={"ID":"3324d3b7-d44b-4905-bf8c-6346891a7189","Type":"ContainerStarted","Data":"81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254"} Oct 31 00:16:54.710163 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.710077 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" event={"ID":"3324d3b7-d44b-4905-bf8c-6346891a7189","Type":"ContainerStarted","Data":"69b694dcad3d857b645e90e9eb08783bf5e278fbbff248ef26d6ff971c5996e6"} Oct 31 00:16:54.710237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.710163 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:16:54.823417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.823362 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" podStartSLOduration=1.823345283 podStartE2EDuration="1.823345283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:54.822766355 +0000 UTC m=+794.916056556" watchObservedRunningTime="2026-10-31 00:16:54.823345283 +0000 UTC m=+794.916635470" Oct 31 00:16:54.939305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:54.939235 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" podStartSLOduration=1.939217371 podStartE2EDuration="1.939217371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:54.93762196 +0000 UTC m=+795.030912168" watchObservedRunningTime="2026-10-31 00:16:54.939217371 +0000 UTC m=+795.032507560" Oct 31 00:16:56.084524 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:56.084493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:56.243986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:56.243935 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:56.512649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:56.512622 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:56.669856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:16:56.669829 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:16:57.475737 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:16:57.475704 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:17:00.978531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:00.978499 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:01.176598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:01.176560 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:01.345057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:01.345016 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:04.959071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:04.959037 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:17:05.720888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:05.720861 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:17:05.721081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:05.720929 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:17:05.954744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:05.954711 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:10.770595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:10.770560 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:13.150657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:13.150624 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:17:15.642070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:15.642028 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:16.867298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.867267 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz"] Oct 31 00:17:16.874778 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.874758 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:16.932079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.932039 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:16.932280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.932091 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxvf\" (UniqueName: \"kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:16.932339 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.932305 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:16.977373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.977333 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:17:16.977373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.977340 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:17:16.977373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.977345 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:17:16.977592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.977340 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:17:16.978829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:16.978810 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz"] Oct 31 00:17:17.033522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.033485 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.033719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.033582 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.033719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.033611 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxvf\" (UniqueName: \"kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.034375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.034350 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.036137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.036119 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.094815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.094782 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxvf\" (UniqueName: \"kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf\") pod \"maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.213863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.213767 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:17.587554 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.587525 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz"] Oct 31 00:17:17.590659 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:17:17.590632 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4a1f8a_f866_4f08_93d1_321a32e69a8d.slice/crio-733afbf665dfdb2657cdd5e96bfc9662e1d2f486543c7a1a0d4985ab4a1efb6e WatchSource:0}: Error finding container 733afbf665dfdb2657cdd5e96bfc9662e1d2f486543c7a1a0d4985ab4a1efb6e: Status 404 returned error can't find the container with id 733afbf665dfdb2657cdd5e96bfc9662e1d2f486543c7a1a0d4985ab4a1efb6e Oct 31 00:17:17.825757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:17.825723 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" event={"ID":"5e4a1f8a-f866-4f08-93d1-321a32e69a8d","Type":"ContainerStarted","Data":"733afbf665dfdb2657cdd5e96bfc9662e1d2f486543c7a1a0d4985ab4a1efb6e"} Oct 31 00:17:21.085135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:21.085049 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:21.848596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:21.848557 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" event={"ID":"5e4a1f8a-f866-4f08-93d1-321a32e69a8d","Type":"ContainerStarted","Data":"126026312cf2de05ce5b85b08d6fe5f912c69a29963e27f93062998a3340ca26"} Oct 31 00:17:21.965229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:21.965126 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" podStartSLOduration=2.431576985 podStartE2EDuration="5.965110609s" totalImagesPullingTime="3.533533624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:21.963462529 +0000 UTC m=+822.056752736" watchObservedRunningTime="2026-10-31 00:17:21.965110609 +0000 UTC m=+822.058400796" Oct 31 00:17:22.855605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:22.855572 5943 generic.go:356] "Generic (PLEG): container finished" podID="5e4a1f8a-f866-4f08-93d1-321a32e69a8d" containerID="126026312cf2de05ce5b85b08d6fe5f912c69a29963e27f93062998a3340ca26" exitCode=0 Oct 31 00:17:22.856001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:22.855615 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" event={"ID":"5e4a1f8a-f866-4f08-93d1-321a32e69a8d","Type":"ContainerDied","Data":"126026312cf2de05ce5b85b08d6fe5f912c69a29963e27f93062998a3340ca26"} Oct 31 00:17:23.999156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:23.999133 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:24.007385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.007368 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf\" (UniqueName: \"kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf\") pod \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " Oct 31 00:17:24.007482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.007457 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca\") pod \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " Oct 31 00:17:24.007531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.007516 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca\") pod \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\" (UID: \"5e4a1f8a-f866-4f08-93d1-321a32e69a8d\") " Oct 31 00:17:24.007902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.007877 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca" pod "5e4a1f8a-f866-4f08-93d1-321a32e69a8d" (UID: "5e4a1f8a-f866-4f08-93d1-321a32e69a8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:24.009653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.009631 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca" pod "5e4a1f8a-f866-4f08-93d1-321a32e69a8d" (UID: "5e4a1f8a-f866-4f08-93d1-321a32e69a8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:24.009739 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.009698 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf" pod "5e4a1f8a-f866-4f08-93d1-321a32e69a8d" (UID: "5e4a1f8a-f866-4f08-93d1-321a32e69a8d"). InnerVolumeSpecName "kube-api-access-svxvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:24.108969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.108936 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:24.108969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.108965 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:24.108969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.108976 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svxvf\" (UniqueName: \"kubernetes.io/projected/5e4a1f8a-f866-4f08-93d1-321a32e69a8d-kube-api-access-svxvf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:24.867303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.867269 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" event={"ID":"5e4a1f8a-f866-4f08-93d1-321a32e69a8d","Type":"ContainerDied","Data":"733afbf665dfdb2657cdd5e96bfc9662e1d2f486543c7a1a0d4985ab4a1efb6e"} Oct 31 00:17:24.867303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.867302 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz" Oct 31 00:17:24.867509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:24.867323 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733afbf665dfdb2657cdd5e96bfc9662e1d2f486543c7a1a0d4985ab4a1efb6e" Oct 31 00:17:26.282935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:26.282899 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:27.475346 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:17:27.475308 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:17:27.968687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:27.968659 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf"] Oct 31 00:17:28.335002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:28.334974 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:28.479276 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:28.479237 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:30.863126 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:30.863092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:31.016319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:31.016288 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:31.436770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:31.436736 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:32.969258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:32.969222 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" podUID="b9fd454f-fef4-4b8e-8072-5417811ad308" containerName="payload-processing" containerID="cri-o://d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6" gracePeriod=25 Oct 31 00:17:33.231786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.231724 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:17:33.301731 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.301698 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf\" (UniqueName: \"kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf\") pod \"b9fd454f-fef4-4b8e-8072-5417811ad308\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " Oct 31 00:17:33.301951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.301900 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume\") pod \"b9fd454f-fef4-4b8e-8072-5417811ad308\" (UID: \"b9fd454f-fef4-4b8e-8072-5417811ad308\") " Oct 31 00:17:33.302302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.302275 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume" pod "b9fd454f-fef4-4b8e-8072-5417811ad308" (UID: "b9fd454f-fef4-4b8e-8072-5417811ad308"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:33.304170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.304145 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf" pod "b9fd454f-fef4-4b8e-8072-5417811ad308" (UID: "b9fd454f-fef4-4b8e-8072-5417811ad308"). InnerVolumeSpecName "kube-api-access-dglhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:33.403136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.403098 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fd454f-fef4-4b8e-8072-5417811ad308-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:33.403136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.403127 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglhf\" (UniqueName: \"kubernetes.io/projected/b9fd454f-fef4-4b8e-8072-5417811ad308-kube-api-access-dglhf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:33.911735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.911695 5943 generic.go:356] "Generic (PLEG): container finished" podID="b9fd454f-fef4-4b8e-8072-5417811ad308" containerID="d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6" exitCode=0 Oct 31 00:17:33.911942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.911746 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" Oct 31 00:17:33.911942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.911778 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" event={"ID":"b9fd454f-fef4-4b8e-8072-5417811ad308","Type":"ContainerDied","Data":"d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6"} Oct 31 00:17:33.911942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.911866 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf" event={"ID":"b9fd454f-fef4-4b8e-8072-5417811ad308","Type":"ContainerDied","Data":"c5ff1e6bf6300a836a6023cdc5a5f08d755c832fa27f9329928e445d0aca1996"} Oct 31 00:17:33.911942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.911899 5943 scope.go:122] "RemoveContainer" containerID="d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6" Oct 31 00:17:33.920944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.920929 5943 scope.go:122] "RemoveContainer" containerID="d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6" Oct 31 00:17:33.921230 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:17:33.921211 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6\": container with ID starting with d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6 not found: ID does not exist" containerID="d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6" Oct 31 00:17:33.921289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:33.921241 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6"} err="failed to get container status \"d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6\": rpc error: code = NotFound desc = could not find container \"d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6\": container with ID starting with d96b4a7d13b9e9f4fdb38490951be4d71eadc27279414104ebbaaef43a5e07f6 not found: ID does not exist" Oct 31 00:17:34.074680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:34.074593 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf"] Oct 31 00:17:34.124512 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:34.124469 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e1bbe4b-84f54fc89-j9jrf"] Oct 31 00:17:35.610239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:35.610203 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:36.041907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:36.041873 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fd454f-fef4-4b8e-8072-5417811ad308" path="/var/lib/kubelet/pods/b9fd454f-fef4-4b8e-8072-5417811ad308/volumes" Oct 31 00:17:37.106216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:37.105978 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:37.256410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:37.256354 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:37.604944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:37.604915 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:37.939145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:37.939065 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:38.160654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.160616 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz"] Oct 31 00:17:38.209982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.209896 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e1bbe4b-s8rlz"] Oct 31 00:17:38.363962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.363926 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:38.593572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.593539 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6"] Oct 31 00:17:38.593998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.593985 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5e4a1f8a-f866-4f08-93d1-321a32e69a8d" containerName="revoke-keys" Oct 31 00:17:38.594045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.594000 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4a1f8a-f866-4f08-93d1-321a32e69a8d" containerName="revoke-keys" Oct 31 00:17:38.594045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.594020 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b9fd454f-fef4-4b8e-8072-5417811ad308" containerName="payload-processing" Oct 31 00:17:38.594045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.594026 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fd454f-fef4-4b8e-8072-5417811ad308" containerName="payload-processing" Oct 31 00:17:38.594140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.594082 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4a1f8a-f866-4f08-93d1-321a32e69a8d" containerName="revoke-keys" Oct 31 00:17:38.594140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.594092 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fd454f-fef4-4b8e-8072-5417811ad308" containerName="payload-processing" Oct 31 00:17:38.598468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.598444 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.650592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.650532 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:17:38.650592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.650581 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:17:38.651153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.651128 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgz2\" (UniqueName: \"kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.651306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.651219 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.651306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.651248 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.701571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.701537 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6"] Oct 31 00:17:38.703793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.703768 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:17:38.703952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.703906 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:17:38.752630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.752601 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgz2\" (UniqueName: \"kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.752810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.752636 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.752810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.752659 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.753396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.753370 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.755078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.755062 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.814247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.814210 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgz2\" (UniqueName: \"kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2\") pod \"maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:38.910927 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:38.910836 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:39.157589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.157556 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:39.283209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.283158 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6"] Oct 31 00:17:39.285904 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:17:39.285872 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9dcc43_f174_426e_9350_ebbfb3909046.slice/crio-c9a2583f3b2c4c5ead32f4635f62b37c67b241ab5952ada7a733cd7d5a4ebf45 WatchSource:0}: Error finding container c9a2583f3b2c4c5ead32f4635f62b37c67b241ab5952ada7a733cd7d5a4ebf45: Status 404 returned error can't find the container with id c9a2583f3b2c4c5ead32f4635f62b37c67b241ab5952ada7a733cd7d5a4ebf45 Oct 31 00:17:39.434754 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.434719 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:39.800655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.800562 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:39.944716 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.944677 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" event={"ID":"6d9dcc43-f174-426e-9350-ebbfb3909046","Type":"ContainerStarted","Data":"8b57cfaeaa1da63aed065e0d105b07e61aee9b6c9cab94ce0d52872773052417"} Oct 31 00:17:39.944881 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.944757 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" event={"ID":"6d9dcc43-f174-426e-9350-ebbfb3909046","Type":"ContainerStarted","Data":"c9a2583f3b2c4c5ead32f4635f62b37c67b241ab5952ada7a733cd7d5a4ebf45"} Oct 31 00:17:39.948646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:39.948616 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.042958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.042929 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4a1f8a-f866-4f08-93d1-321a32e69a8d" path="/var/lib/kubelet/pods/5e4a1f8a-f866-4f08-93d1-321a32e69a8d/volumes" Oct 31 00:17:40.060468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.060356 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" podStartSLOduration=2.060338766 podStartE2EDuration="2.060338766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:40.05916224 +0000 UTC m=+840.152452450" watchObservedRunningTime="2026-10-31 00:17:40.060338766 +0000 UTC m=+840.153628954" Oct 31 00:17:40.229488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.229451 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.381658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.381564 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.539557 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.539510 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.695340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.695251 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.951147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.951065 5943 generic.go:356] "Generic (PLEG): container finished" podID="6d9dcc43-f174-426e-9350-ebbfb3909046" containerID="8b57cfaeaa1da63aed065e0d105b07e61aee9b6c9cab94ce0d52872773052417" exitCode=0 Oct 31 00:17:40.951319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:40.951152 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" event={"ID":"6d9dcc43-f174-426e-9350-ebbfb3909046","Type":"ContainerDied","Data":"8b57cfaeaa1da63aed065e0d105b07e61aee9b6c9cab94ce0d52872773052417"} Oct 31 00:17:41.263233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:41.263203 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:41.422355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:41.422324 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:41.573511 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:41.573421 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:41.741351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:41.741312 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:41.891882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:41.891795 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:42.095118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.095094 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:42.185325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.185244 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca\") pod \"6d9dcc43-f174-426e-9350-ebbfb3909046\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " Oct 31 00:17:42.185325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.185326 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2\" (UniqueName: \"kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2\") pod \"6d9dcc43-f174-426e-9350-ebbfb3909046\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " Oct 31 00:17:42.185571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.185387 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca\") pod \"6d9dcc43-f174-426e-9350-ebbfb3909046\" (UID: \"6d9dcc43-f174-426e-9350-ebbfb3909046\") " Oct 31 00:17:42.185632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.185582 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca" pod "6d9dcc43-f174-426e-9350-ebbfb3909046" (UID: "6d9dcc43-f174-426e-9350-ebbfb3909046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:42.185728 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.185705 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:42.187701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.187673 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2" pod "6d9dcc43-f174-426e-9350-ebbfb3909046" (UID: "6d9dcc43-f174-426e-9350-ebbfb3909046"). InnerVolumeSpecName "kube-api-access-9mgz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:42.187804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.187746 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca" pod "6d9dcc43-f174-426e-9350-ebbfb3909046" (UID: "6d9dcc43-f174-426e-9350-ebbfb3909046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:42.208764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.208740 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:42.286652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.286621 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgz2\" (UniqueName: \"kubernetes.io/projected/6d9dcc43-f174-426e-9350-ebbfb3909046-kube-api-access-9mgz2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:42.286652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.286650 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9dcc43-f174-426e-9350-ebbfb3909046-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:42.653753 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.653715 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:42.964395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.964321 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" Oct 31 00:17:42.964536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.964323 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6" event={"ID":"6d9dcc43-f174-426e-9350-ebbfb3909046","Type":"ContainerDied","Data":"c9a2583f3b2c4c5ead32f4635f62b37c67b241ab5952ada7a733cd7d5a4ebf45"} Oct 31 00:17:42.964536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:42.964468 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a2583f3b2c4c5ead32f4635f62b37c67b241ab5952ada7a733cd7d5a4ebf45" Oct 31 00:17:43.150997 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.150955 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:17:43.327293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.327261 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww"] Oct 31 00:17:43.327695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.327682 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6d9dcc43-f174-426e-9350-ebbfb3909046" containerName="revoke-keys" Oct 31 00:17:43.327742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.327697 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9dcc43-f174-426e-9350-ebbfb3909046" containerName="revoke-keys" Oct 31 00:17:43.327803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.327794 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9dcc43-f174-426e-9350-ebbfb3909046" containerName="revoke-keys" Oct 31 00:17:43.331026 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.330991 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.393342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.393310 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww"] Oct 31 00:17:43.399607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399575 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.399782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399621 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.399782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399660 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.399950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399831 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.399950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399856 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzpf\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.399950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399878 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.400105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399955 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.400105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.399979 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.400105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.400004 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.415413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.415382 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:43.458752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.458721 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-10a7aa-openshift-default-dockercfg-zqppq\"" Oct 31 00:17:43.500936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.500902 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.500940 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzpf\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.500964 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501016 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501035 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501062 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501138 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501163 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501233 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501537 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501622 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501580 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501642 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.501756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.501733 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.502147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.502128 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.503548 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.503525 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.503987 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.503970 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.562259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.562226 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzpf\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.562259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.562248 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.643845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.643758 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:43.662676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:43.662646 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.074494 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:44.074471 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww"] Oct 31 00:17:44.076284 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:17:44.076261 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode346dc38_bb4a_41ec_80a4_5f9fd9478e34.slice/crio-ec2dddf6820b96331784e9d308d559a657a0fda16b39b9fbdf0cf007b0d636c5 WatchSource:0}: Error finding container ec2dddf6820b96331784e9d308d559a657a0fda16b39b9fbdf0cf007b0d636c5: Status 404 returned error can't find the container with id ec2dddf6820b96331784e9d308d559a657a0fda16b39b9fbdf0cf007b0d636c5 Oct 31 00:17:44.078683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:44.078652 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:17:44.078786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:44.078729 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:17:44.078786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:44.078775 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:17:44.975497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:44.975455 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" event={"ID":"e346dc38-bb4a-41ec-80a4-5f9fd9478e34","Type":"ContainerStarted","Data":"28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe"} Oct 31 00:17:44.975873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:44.975516 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" event={"ID":"e346dc38-bb4a-41ec-80a4-5f9fd9478e34","Type":"ContainerStarted","Data":"ec2dddf6820b96331784e9d308d559a657a0fda16b39b9fbdf0cf007b0d636c5"} Oct 31 00:17:45.194037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.193975 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podStartSLOduration=2.193957606 podStartE2EDuration="2.193957606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:45.144660278 +0000 UTC m=+845.237950467" watchObservedRunningTime="2026-10-31 00:17:45.193957606 +0000 UTC m=+845.287247792" Oct 31 00:17:45.615402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.615365 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:45.644889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.644853 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:45.650601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.650576 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:45.891660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.891559 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:45.980558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.980527 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:45.981676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:45.981656 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:17:46.046885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:46.046853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:46.330579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:46.330540 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:46.530824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:46.530782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:46.735632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:46.735529 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:48.969496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:48.969462 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:49.143462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:49.143423 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.447358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:50.447319 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.640111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:50.640068 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.784922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:50.784893 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:50.934473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:50.934439 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:51.085498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:51.085417 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:55.950282 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:55.950248 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:56.249109 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:56.249026 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:56.430739 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:56.430700 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:57.387286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:57.387257 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx"] Oct 31 00:17:57.475291 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:17:57.475254 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:17:57.496286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:57.496256 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl"] Oct 31 00:17:57.751370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:57.751279 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:57.918562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:17:57.918509 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:00.728219 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:00.728170 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:00.926830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:00.926795 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:01.069496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:01.069460 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:01.262530 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:01.262497 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:01.443706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:01.443612 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:02.387981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.387934 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" podUID="3324d3b7-d44b-4905-bf8c-6346891a7189" containerName="payload-processing" containerID="cri-o://81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254" gracePeriod=25 Oct 31 00:18:02.497438 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.497381 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" podUID="3cf82392-9e81-4e0d-9568-c2d37cd622cc" containerName="payload-pre-processing" containerID="cri-o://f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9" gracePeriod=25 Oct 31 00:18:02.659280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.659245 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:18:02.699918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.699890 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume\") pod \"3324d3b7-d44b-4905-bf8c-6346891a7189\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " Oct 31 00:18:02.700107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.700042 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n\" (UniqueName: \"kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n\") pod \"3324d3b7-d44b-4905-bf8c-6346891a7189\" (UID: \"3324d3b7-d44b-4905-bf8c-6346891a7189\") " Oct 31 00:18:02.700345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.700321 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume" pod "3324d3b7-d44b-4905-bf8c-6346891a7189" (UID: "3324d3b7-d44b-4905-bf8c-6346891a7189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:02.700444 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.700423 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3324d3b7-d44b-4905-bf8c-6346891a7189-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:02.702361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.702341 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n" pod "3324d3b7-d44b-4905-bf8c-6346891a7189" (UID: "3324d3b7-d44b-4905-bf8c-6346891a7189"). InnerVolumeSpecName "kube-api-access-9gg8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:02.762276 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.762249 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:18:02.801277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.801239 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume\") pod \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " Oct 31 00:18:02.801476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.801355 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp\" (UniqueName: \"kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp\") pod \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\" (UID: \"3cf82392-9e81-4e0d-9568-c2d37cd622cc\") " Oct 31 00:18:02.801676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.801655 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gg8n\" (UniqueName: \"kubernetes.io/projected/3324d3b7-d44b-4905-bf8c-6346891a7189-kube-api-access-9gg8n\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:02.801751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.801711 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume" pod "3cf82392-9e81-4e0d-9568-c2d37cd622cc" (UID: "3cf82392-9e81-4e0d-9568-c2d37cd622cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:02.803565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.803543 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp" pod "3cf82392-9e81-4e0d-9568-c2d37cd622cc" (UID: "3cf82392-9e81-4e0d-9568-c2d37cd622cc"). InnerVolumeSpecName "kube-api-access-p5hvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:02.902935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.902866 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5hvp\" (UniqueName: \"kubernetes.io/projected/3cf82392-9e81-4e0d-9568-c2d37cd622cc-kube-api-access-p5hvp\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:02.902935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:02.902891 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf82392-9e81-4e0d-9568-c2d37cd622cc-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:03.064337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.064302 5943 generic.go:356] "Generic (PLEG): container finished" podID="3cf82392-9e81-4e0d-9568-c2d37cd622cc" containerID="f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9" exitCode=0 Oct 31 00:18:03.064530 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.064349 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" Oct 31 00:18:03.064530 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.064382 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" event={"ID":"3cf82392-9e81-4e0d-9568-c2d37cd622cc","Type":"ContainerDied","Data":"f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9"} Oct 31 00:18:03.064530 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.064442 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl" event={"ID":"3cf82392-9e81-4e0d-9568-c2d37cd622cc","Type":"ContainerDied","Data":"0bb86ea21592f3193f35cb5a1a163f75736cdf32265cce62d0de913b6abc1e8d"} Oct 31 00:18:03.064530 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.064473 5943 scope.go:122] "RemoveContainer" containerID="f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9" Oct 31 00:18:03.065884 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.065864 5943 generic.go:356] "Generic (PLEG): container finished" podID="3324d3b7-d44b-4905-bf8c-6346891a7189" containerID="81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254" exitCode=0 Oct 31 00:18:03.065973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.065897 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" event={"ID":"3324d3b7-d44b-4905-bf8c-6346891a7189","Type":"ContainerDied","Data":"81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254"} Oct 31 00:18:03.065973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.065907 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" Oct 31 00:18:03.065973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.065941 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx" event={"ID":"3324d3b7-d44b-4905-bf8c-6346891a7189","Type":"ContainerDied","Data":"69b694dcad3d857b645e90e9eb08783bf5e278fbbff248ef26d6ff971c5996e6"} Oct 31 00:18:03.073969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.073952 5943 scope.go:122] "RemoveContainer" containerID="f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9" Oct 31 00:18:03.074276 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:18:03.074258 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9\": container with ID starting with f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9 not found: ID does not exist" containerID="f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9" Oct 31 00:18:03.074338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.074284 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9"} err="failed to get container status \"f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9\": rpc error: code = NotFound desc = could not find container \"f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9\": container with ID starting with f629a2262686a8736887cf904879435cdea403d408993af34960d066fba7f6d9 not found: ID does not exist" Oct 31 00:18:03.074338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.074300 5943 scope.go:122] "RemoveContainer" containerID="81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254" Oct 31 00:18:03.082426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.082410 5943 scope.go:122] "RemoveContainer" containerID="81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254" Oct 31 00:18:03.082680 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:18:03.082661 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254\": container with ID starting with 81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254 not found: ID does not exist" containerID="81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254" Oct 31 00:18:03.082721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.082688 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254"} err="failed to get container status \"81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254\": rpc error: code = NotFound desc = could not find container \"81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254\": container with ID starting with 81208a76710b2f9c647b4237c79f92c7286b78ecbd59d65fc322796932c17254 not found: ID does not exist" Oct 31 00:18:03.227601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.227522 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl"] Oct 31 00:18:03.233686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.233655 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83a178e-54cb59d6cf-zhxjl"] Oct 31 00:18:03.340459 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.340432 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx"] Oct 31 00:18:03.344057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:03.344036 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83a178e-57bf8bbb77-drbvx"] Oct 31 00:18:04.039955 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:04.039921 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3324d3b7-d44b-4905-bf8c-6346891a7189" path="/var/lib/kubelet/pods/3324d3b7-d44b-4905-bf8c-6346891a7189/volumes" Oct 31 00:18:04.040616 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:04.040469 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf82392-9e81-4e0d-9568-c2d37cd622cc" path="/var/lib/kubelet/pods/3cf82392-9e81-4e0d-9568-c2d37cd622cc/volumes" Oct 31 00:18:05.577954 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:05.577895 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:05.739891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:05.739853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:05.887973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:05.887890 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:09.061347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:09.061317 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:18:09.908735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:09.908691 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:10.716486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:10.716446 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:10.909369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:10.909336 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:11.463808 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:11.463772 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:11.656786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:11.656748 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:11.801816 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:11.801779 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:12.073006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:12.072920 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:12.223134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:12.223092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:12.365950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:12.365855 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:12.904351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:12.904316 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:13.543135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:13.543099 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:13.981846 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:13.981757 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:15.381635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:15.381603 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:18:16.318482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:16.318449 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:18.012663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.012633 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:18.277385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.277310 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:18.420509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.420471 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:18.548014 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.547922 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6"] Oct 31 00:18:18.597482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.597450 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b83a178e-rnnh6"] Oct 31 00:18:18.673341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.673312 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:18.921400 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:18.921315 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:19.091222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:19.091160 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:19.236445 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:19.236357 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:19.486323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:19.486291 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:20.040741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:20.040708 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9dcc43-f174-426e-9350-ebbfb3909046" path="/var/lib/kubelet/pods/6d9dcc43-f174-426e-9350-ebbfb3909046/volumes" Oct 31 00:18:20.213617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:20.213583 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:20.426673 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:20.426593 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:22.188793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:22.188760 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:22.388076 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:22.388033 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:23.588662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:23.588629 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:24.269484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:24.269453 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:24.487240 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:24.487168 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:27.482787 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:18:27.482753 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:18:27.527285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:27.527252 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:27.706218 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:27.706170 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:28.986821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:28.986778 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:31.157129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:31.157092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:31.768273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:31.768233 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:31.774466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:31.774436 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:18:31.961710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:31.961680 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:32.022477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:32.022376 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:32.022477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:32.022460 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:33.218969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:33.218941 5943 generic.go:356] "Generic (PLEG): container finished" podID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerID="28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe" exitCode=0 Oct 31 00:18:33.219394 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:33.219027 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" event={"ID":"e346dc38-bb4a-41ec-80a4-5f9fd9478e34","Type":"ContainerDied","Data":"28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe"} Oct 31 00:18:33.219445 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:33.219433 5943 scope.go:122] "RemoveContainer" containerID="28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe" Oct 31 00:18:33.220652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:33.220625 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:18:33.220727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:33.220695 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:18:33.220766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:33.220732 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:18:34.226435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:34.226399 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" event={"ID":"e346dc38-bb4a-41ec-80a4-5f9fd9478e34","Type":"ContainerStarted","Data":"0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53"} Oct 31 00:18:34.254796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:34.254754 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:34.645646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:34.645604 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:18:34.645889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:34.645741 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:18:34.645889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:34.645804 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:18:35.199815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.199780 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx"] Oct 31 00:18:35.200261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.200242 5943 cpu_manager.go:414] "containerMap: removing container" podUID="3324d3b7-d44b-4905-bf8c-6346891a7189" containerName="payload-processing" Oct 31 00:18:35.200261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.200262 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3324d3b7-d44b-4905-bf8c-6346891a7189" containerName="payload-processing" Oct 31 00:18:35.200423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.200283 5943 cpu_manager.go:414] "containerMap: removing container" podUID="3cf82392-9e81-4e0d-9568-c2d37cd622cc" containerName="payload-pre-processing" Oct 31 00:18:35.200423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.200289 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf82392-9e81-4e0d-9568-c2d37cd622cc" containerName="payload-pre-processing" Oct 31 00:18:35.200423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.200420 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="3324d3b7-d44b-4905-bf8c-6346891a7189" containerName="payload-processing" Oct 31 00:18:35.200581 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.200434 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf82392-9e81-4e0d-9568-c2d37cd622cc" containerName="payload-pre-processing" Oct 31 00:18:35.202650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.202631 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.257866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.257818 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mdg\" (UniqueName: \"kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.258404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.257993 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.258404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.258104 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.258404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.258280 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.258404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.258344 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.258593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.258423 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.300399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.300364 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:35.308958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.308940 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"default-dockercfg-zhdv9\"" Oct 31 00:18:35.309119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.309100 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"simulator-w3-7abb95-kserve-self-signed-certs\"" Oct 31 00:18:35.309212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.309107 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"kube-root-ca.crt\"" Oct 31 00:18:35.309212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.309151 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"openshift-service-ca.crt\"" Oct 31 00:18:35.310604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.310586 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx"] Oct 31 00:18:35.359120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359082 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.359371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359209 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.359371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359273 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.359371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359300 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.359542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359382 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mdg\" (UniqueName: \"kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.359542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359439 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.359776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.359746 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.360060 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.360033 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.360441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.360413 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.361991 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.361968 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.362149 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.362132 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.420372 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.420331 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mdg\" (UniqueName: \"kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg\") pod \"simulator-w3-7abb95-kserve-76cf7846d-rqhmx\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.514211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.514163 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:35.656239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.653570 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:18:35.656239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.653631 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:18:35.917055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:35.916906 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx"] Oct 31 00:18:35.919771 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:18:35.919742 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84856bbd_a4ca_4463_9e0b_6f371f20ee61.slice/crio-58c973a0a3a870d390e659d65717d76995c3aca4591986727231b72fa4cedad4 WatchSource:0}: Error finding container 58c973a0a3a870d390e659d65717d76995c3aca4591986727231b72fa4cedad4: Status 404 returned error can't find the container with id 58c973a0a3a870d390e659d65717d76995c3aca4591986727231b72fa4cedad4 Oct 31 00:18:36.267161 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:36.267118 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" event={"ID":"84856bbd-a4ca-4463-9e0b-6f371f20ee61","Type":"ContainerStarted","Data":"58c973a0a3a870d390e659d65717d76995c3aca4591986727231b72fa4cedad4"} Oct 31 00:18:36.646691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:36.646597 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:18:36.647020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:36.646986 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:18:36.854315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:36.854271 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:37.455864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:37.455045 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:37.612826 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:37.612787 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:37.645116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:37.644630 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:18:37.645116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:37.644699 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:18:37.760199 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:37.760147 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:38.651261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:38.650595 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:18:38.651261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:38.650653 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:18:39.289490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.289448 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" event={"ID":"84856bbd-a4ca-4463-9e0b-6f371f20ee61","Type":"ContainerStarted","Data":"8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b"} Oct 31 00:18:39.395654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.395618 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:39.406053 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.406003 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" podStartSLOduration=2.025532278 podStartE2EDuration="4.40599007s" totalImagesPullingTime="2.380457792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:39.404972254 +0000 UTC m=+899.498262452" watchObservedRunningTime="2026-10-31 00:18:39.40599007 +0000 UTC m=+899.499280257" Oct 31 00:18:39.514687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.514649 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:39.596257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.596138 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:39.647820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.647781 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:18:39.648086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.648065 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:18:39.648799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:39.648781 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 00:18:42.936907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:42.936871 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:43.879573 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:43.879538 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:44.499023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:44.498987 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:44.641708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:44.641670 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:44.789933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:44.789893 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:45.514975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:45.514942 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:45.528209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:45.528168 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:46.333655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:46.333628 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 00:18:47.652783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:47.652748 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:18:49.579768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:49.579692 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:49.772702 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:49.772662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:49.917685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:49.917590 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.638081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:54.638044 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.475865 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:18:57.475818 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:18:57.650870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:57.650831 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:18:59.446348 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:18:59.446309 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.395370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.395336 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg"] Oct 31 00:19:01.399362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.399340 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.502202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.502148 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.503592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.503574 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"simulator-w4-0144bd-kserve-self-signed-certs\"" Oct 31 00:19:01.503666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.503648 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"default-dockercfg-g9zff\"" Oct 31 00:19:01.504745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.504731 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"openshift-service-ca.crt\"" Oct 31 00:19:01.505033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.505015 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg"] Oct 31 00:19:01.506630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.506611 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"kube-root-ca.crt\"" Oct 31 00:19:01.537368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.537316 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.537558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.537428 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.537558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.537507 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.537681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.537624 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsjg\" (UniqueName: \"kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.537681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.537668 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.537785 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.537760 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.638848 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.638804 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsjg\" (UniqueName: \"kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.638848 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.638845 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.638885 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.638946 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.638979 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.639032 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.639574 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.639582 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.639872 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.639745 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.641754 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.641722 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.641901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.641882 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.700344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.700266 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsjg\" (UniqueName: \"kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg\") pod \"simulator-w4-0144bd-kserve-59bd5b7d68-nkplg\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:01.710504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:01.710472 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:02.095147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:02.095122 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg"] Oct 31 00:19:02.097618 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:19:02.097591 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88533ac8_d33e_4d1a_a3e2_b47d0da5f2bc.slice/crio-98ead338aefdefde75bff6d48375c112dd6765d3a00caaef1191cfa589aa7763 WatchSource:0}: Error finding container 98ead338aefdefde75bff6d48375c112dd6765d3a00caaef1191cfa589aa7763: Status 404 returned error can't find the container with id 98ead338aefdefde75bff6d48375c112dd6765d3a00caaef1191cfa589aa7763 Oct 31 00:19:02.403662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:02.403568 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" event={"ID":"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc","Type":"ContainerStarted","Data":"d1c464cc9c0de012812c939438b573b01b5689965046e52ce4dee6b546098f2b"} Oct 31 00:19:02.403662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:02.403614 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" event={"ID":"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc","Type":"ContainerStarted","Data":"98ead338aefdefde75bff6d48375c112dd6765d3a00caaef1191cfa589aa7763"} Oct 31 00:19:02.521668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:02.521614 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podStartSLOduration=1.521599208 podStartE2EDuration="1.521599208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:02.520704889 +0000 UTC m=+922.613995076" watchObservedRunningTime="2026-10-31 00:19:02.521599208 +0000 UTC m=+922.614889394" Oct 31 00:19:02.710960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:02.710872 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:04.336745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:04.336706 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.087506 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:06.087470 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:19:06.809475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:06.809436 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:09.147642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:09.147602 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:11.711303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:11.711268 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:11.724101 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:11.724074 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:12.463403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:12.463370 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:19:13.948546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:13.948509 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.146507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:19.146472 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.395749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:19.395718 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.785833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:19.785800 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:19:19.858789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:19.858752 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:20.010627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:20.010595 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:20.330316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:20.330279 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:21.682988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.682951 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n"] Oct 31 00:19:21.686559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.686539 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.737562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.737525 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.737786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.737658 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.737786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.737707 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.737907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.737856 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.737907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.737895 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl88k\" (UniqueName: \"kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.737996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.737935 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.787985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.787948 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"premium-w3-7abb95-kserve-self-signed-certs\"" Oct 31 00:19:21.789496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.789473 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n"] Oct 31 00:19:21.807437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.807403 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:21.838927 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.838898 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.839129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.839008 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.839129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.839058 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.839129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.839081 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.839334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.839164 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.839334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.839223 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl88k\" (UniqueName: \"kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.840057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.840029 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.840541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.839834 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.840541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.840468 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.841626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.841604 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.842296 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.842266 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.900035 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.900006 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl88k\" (UniqueName: \"kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k\") pod \"premium-w3-7abb95-kserve-f6847976f-qmd6n\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:21.999327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:21.999236 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:22.381590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:22.381565 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n"] Oct 31 00:19:22.502894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:22.502860 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" event={"ID":"2cca0779-c8e3-4252-86f7-d2ece966c43c","Type":"ContainerStarted","Data":"77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9"} Oct 31 00:19:22.503072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:22.502903 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" event={"ID":"2cca0779-c8e3-4252-86f7-d2ece966c43c","Type":"ContainerStarted","Data":"bcc5c80465e53f39d42fe261c6e11d4f1a978f2134b6c8c778af1e69c53f2877"} Oct 31 00:19:22.619616 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:22.619564 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podStartSLOduration=1.619549878 podStartE2EDuration="1.619549878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:22.61808206 +0000 UTC m=+942.711372271" watchObservedRunningTime="2026-10-31 00:19:22.619549878 +0000 UTC m=+942.712840065" Oct 31 00:19:23.000318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:23.000285 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:24.045747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:24.045708 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.229694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:25.229650 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.881429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:25.881398 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.057082 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:26.056897 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:27.476441 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:19:27.476392 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:19:29.152909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:29.152864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:32.000002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:31.999969 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:32.013236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:32.013205 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:32.567517 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:32.567490 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:19:33.812107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:33.812070 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:19:33.966337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:33.966305 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:38.759255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:38.759218 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:42.850691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:42.850656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:43.285999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:43.285962 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:43.612150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:43.612062 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:48.444487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:48.444445 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:49.440219 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:49.440165 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:19:53.355106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.355069 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:53.712412 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.712319 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf"] Oct 31 00:19:53.716129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.716105 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.762525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.762486 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.762725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.762569 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.762725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.762640 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.762725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.762684 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.762725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.762710 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gb5c\" (UniqueName: \"kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.762951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.762741 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.816288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.816258 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:53.817572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.817550 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"premium-w4-0144bd-kserve-self-signed-certs\"" Oct 31 00:19:53.819148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.819129 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf"] Oct 31 00:19:53.864218 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864188 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864390 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864261 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864390 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864296 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864390 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864318 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gb5c\" (UniqueName: \"kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864390 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864363 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864394 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864544 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864769 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.864799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.864785 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.866867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.866841 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.867351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.867331 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:53.924942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:53.924908 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gb5c\" (UniqueName: \"kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c\") pod \"premium-w4-0144bd-kserve-bcb9d9c85-2d7rf\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:54.029447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:54.029404 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:54.411254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:54.411230 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf"] Oct 31 00:19:54.413676 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:19:54.413647 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4c15de9_f253_4fa2_88f2_82c201288e2c.slice/crio-c45f43ca80c43ef702e09e4ae10f86ca9350fa398da3e1a36a9cda0fa4307003 WatchSource:0}: Error finding container c45f43ca80c43ef702e09e4ae10f86ca9350fa398da3e1a36a9cda0fa4307003: Status 404 returned error can't find the container with id c45f43ca80c43ef702e09e4ae10f86ca9350fa398da3e1a36a9cda0fa4307003 Oct 31 00:19:54.671975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:54.671888 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" event={"ID":"f4c15de9-f253-4fa2-88f2-82c201288e2c","Type":"ContainerStarted","Data":"e7bdaa28af34e8f27219bef82faf04150d0c6ba61768a26ac7a74705c85b4614"} Oct 31 00:19:54.671975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:54.671927 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" event={"ID":"f4c15de9-f253-4fa2-88f2-82c201288e2c","Type":"ContainerStarted","Data":"c45f43ca80c43ef702e09e4ae10f86ca9350fa398da3e1a36a9cda0fa4307003"} Oct 31 00:19:54.788357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:54.788304 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" podStartSLOduration=1.788288367 podStartE2EDuration="1.788288367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:54.78707345 +0000 UTC m=+974.880363654" watchObservedRunningTime="2026-10-31 00:19:54.788288367 +0000 UTC m=+974.881578586" Oct 31 00:19:55.030239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:55.030199 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:19:57.476272 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:19:57.476233 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:19:58.166002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:19:58.165964 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:01.415080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.415046 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b"] Oct 31 00:20:01.418819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.418802 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.500975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.500946 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"default-dockercfg-vgb7c\"" Oct 31 00:20:01.536079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.536044 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.536297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.536092 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gks\" (UniqueName: \"kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.536297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.536231 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.536387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.536347 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.536426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.536394 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.536426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.536415 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.550544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.550510 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"simulator-w1-869cca-kserve-self-signed-certs\"" Oct 31 00:20:01.550683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.550571 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"kube-root-ca.crt\"" Oct 31 00:20:01.550683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.550624 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"openshift-service-ca.crt\"" Oct 31 00:20:01.552238 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.552222 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b"] Oct 31 00:20:01.637377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637344 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637390 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637417 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637451 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637482 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gks\" (UniqueName: \"kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637516 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637837 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.637949 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637929 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.638000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.637962 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.639888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.639864 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.640041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.640024 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.642787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.642753 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:01.699137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.699063 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gks\" (UniqueName: \"kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks\") pod \"simulator-w1-869cca-kserve-7ff4785566-gb24b\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:01.731381 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:01.731343 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:02.109098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:02.109074 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b"] Oct 31 00:20:02.112080 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:20:02.112051 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb781bdd4_3456_4da2_aa92_021f64272877.slice/crio-9c60e6127e92655573598e520a38c0b231fc7c5a0b065ec43562aa963f0a5349 WatchSource:0}: Error finding container 9c60e6127e92655573598e520a38c0b231fc7c5a0b065ec43562aa963f0a5349: Status 404 returned error can't find the container with id 9c60e6127e92655573598e520a38c0b231fc7c5a0b065ec43562aa963f0a5349 Oct 31 00:20:02.720208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:02.720163 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" event={"ID":"b781bdd4-3456-4da2-aa92-021f64272877","Type":"ContainerStarted","Data":"cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c"} Oct 31 00:20:02.720592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:02.720218 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" event={"ID":"b781bdd4-3456-4da2-aa92-021f64272877","Type":"ContainerStarted","Data":"9c60e6127e92655573598e520a38c0b231fc7c5a0b065ec43562aa963f0a5349"} Oct 31 00:20:02.732225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:02.732170 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:02.837976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:02.837911 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podStartSLOduration=1.8378964469999999 podStartE2EDuration="1.837896447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:02.836736876 +0000 UTC m=+982.930027080" watchObservedRunningTime="2026-10-31 00:20:02.837896447 +0000 UTC m=+982.931186634" Oct 31 00:20:03.062724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:03.062688 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.030277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:04.030238 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:20:04.043386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:04.043357 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:20:04.743120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:04.743082 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:20:07.863286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:07.863236 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:08.747249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.747215 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7"] Oct 31 00:20:08.751783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.751762 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.807907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.807876 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"premium-w0-10a7aa-kserve-self-signed-certs\"" Oct 31 00:20:08.807907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.807877 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"kube-root-ca.crt\"" Oct 31 00:20:08.815386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.815351 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7x8v\" (UniqueName: \"kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.815525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.815414 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.815525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.815477 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.815623 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.815580 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.815660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.815622 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.815711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.815676 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.857566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.857533 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"default-dockercfg-2clwg\"" Oct 31 00:20:08.859099 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.859078 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7"] Oct 31 00:20:08.876497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.876468 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:08.907721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.907692 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"openshift-service-ca.crt\"" Oct 31 00:20:08.916460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.916417 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.916621 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.916520 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.916689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.916631 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7x8v\" (UniqueName: \"kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.916755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.916701 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.916755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.916724 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.916869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.916756 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.917690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.917660 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.918880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.918850 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.919015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.919001 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.920626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.920600 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.922399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.922224 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:08.978284 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:08.978246 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7x8v\" (UniqueName: \"kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v\") pod \"premium-w0-10a7aa-kserve-cd4d5979-tpcx7\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:09.063837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:09.063803 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:09.441262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:09.441191 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7"] Oct 31 00:20:09.443935 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:20:09.443903 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ff896c2_e62d_4484_826d_526bcd6adc17.slice/crio-85b311fd87be05b8f51dab024e056c7ff6043fe19c1b72d1ed0734fb8fb4fec8 WatchSource:0}: Error finding container 85b311fd87be05b8f51dab024e056c7ff6043fe19c1b72d1ed0734fb8fb4fec8: Status 404 returned error can't find the container with id 85b311fd87be05b8f51dab024e056c7ff6043fe19c1b72d1ed0734fb8fb4fec8 Oct 31 00:20:09.758233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:09.758173 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" event={"ID":"1ff896c2-e62d-4484-826d-526bcd6adc17","Type":"ContainerStarted","Data":"7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519"} Oct 31 00:20:09.758403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:09.758242 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" event={"ID":"1ff896c2-e62d-4484-826d-526bcd6adc17","Type":"ContainerStarted","Data":"85b311fd87be05b8f51dab024e056c7ff6043fe19c1b72d1ed0734fb8fb4fec8"} Oct 31 00:20:09.875594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:09.875542 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podStartSLOduration=1.8755275359999999 podStartE2EDuration="1.875527536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:09.875484644 +0000 UTC m=+989.968774863" watchObservedRunningTime="2026-10-31 00:20:09.875527536 +0000 UTC m=+989.968817721" Oct 31 00:20:10.064733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:10.064651 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:11.732002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:11.731971 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:11.745479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:11.745451 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:11.780174 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:11.780141 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:20:12.664439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:12.664397 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.575326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:17.575287 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:19.064585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:19.064487 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:19.084195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:19.084158 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:19.823036 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:19.823005 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:20:19.973945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:19.973908 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:20:22.356959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:22.356920 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.150600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:27.150570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.477010 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:20:27.476927 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:20:32.159602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:32.159567 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:37.059796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:37.059759 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:41.957050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:41.957012 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:42.232049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:42.231959 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:43.641246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:43.641204 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:46.867543 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:46.867501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:47.328989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:47.328952 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:20:50.807915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:50.807883 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:51.873574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:51.873523 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.272039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:54.272003 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.473874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:54.473821 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.768667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:20:56.768631 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:57.476308 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:20:57.476274 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:21:01.764767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:01.764724 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:06.696743 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:06.696686 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:11.746102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:11.746059 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:12.782015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.781982 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d"] Oct 31 00:21:12.786028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.786006 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.887705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.887673 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"premium-w1-869cca-kserve-self-signed-certs\"" Oct 31 00:21:12.889241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.889210 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:12.889790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.889766 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d"] Oct 31 00:21:12.898328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.898300 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.898427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.898335 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfb6\" (UniqueName: \"kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.898427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.898356 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.898558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.898533 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.898684 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.898666 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.898759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.898724 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999358 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999413 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999540 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfb6\" (UniqueName: \"kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999578 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999780 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999642 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999780 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999760 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999886 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:12.999970 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:12.999946 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:13.000421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:13.000397 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:13.003002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:13.002975 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:13.003131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:13.003116 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:13.061114 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:13.061028 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfb6\" (UniqueName: \"kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6\") pod \"premium-w1-869cca-kserve-f49c566dc-jg47d\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:13.098114 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:13.098076 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:13.495052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:13.495029 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d"] Oct 31 00:21:13.497930 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:21:13.497895 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50c74fcf_a456_45ae_a09d_4cbbe59714ba.slice/crio-acaf1553671790d794b9cda75e86eb7d201890a8eeea296e59d161eb85d2deed WatchSource:0}: Error finding container acaf1553671790d794b9cda75e86eb7d201890a8eeea296e59d161eb85d2deed: Status 404 returned error can't find the container with id acaf1553671790d794b9cda75e86eb7d201890a8eeea296e59d161eb85d2deed Oct 31 00:21:14.096540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:14.096495 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" event={"ID":"50c74fcf-a456-45ae-a09d-4cbbe59714ba","Type":"ContainerStarted","Data":"5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee"} Oct 31 00:21:14.096939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:14.096548 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" event={"ID":"50c74fcf-a456-45ae-a09d-4cbbe59714ba","Type":"ContainerStarted","Data":"acaf1553671790d794b9cda75e86eb7d201890a8eeea296e59d161eb85d2deed"} Oct 31 00:21:14.098865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:14.098849 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:14.217742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:14.217693 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podStartSLOduration=2.217677924 podStartE2EDuration="2.217677924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:14.215296737 +0000 UTC m=+1054.308586925" watchObservedRunningTime="2026-10-31 00:21:14.217677924 +0000 UTC m=+1054.310968111" Oct 31 00:21:16.668678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:16.668645 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:20.290891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:20.290856 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:21:21.527922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:21.527892 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.098804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:23.098761 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:23.111747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:23.111713 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:23.152815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:23.152787 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 00:21:26.268577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:26.268543 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:27.475886 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:21:27.475844 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:21:31.163725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:31.163691 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:32.239417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.239381 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7"] Oct 31 00:21:32.242954 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.242938 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.295371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.295341 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:21:32.307576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.307537 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p9d\" (UniqueName: \"kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.307733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.307587 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.307815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.307729 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.344991 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.344955 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:21:32.344991 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.344963 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:21:32.345254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.344987 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:21:32.347046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.347023 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7"] Oct 31 00:21:32.408409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.408372 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.408609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.408465 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.408609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.408542 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p9d\" (UniqueName: \"kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.409046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.409023 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.410993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.410970 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.469804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.469775 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p9d\" (UniqueName: \"kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d\") pod \"maas-api-revoke-sub-c7011d03-bssc7\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.556040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.555999 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:32.931173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:32.931149 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7"] Oct 31 00:21:32.933608 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:21:32.933582 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af89437_140e_402d_ad56_bda892fda9c8.slice/crio-d07d35960ec75ed4d9a13674aea42dc4dec735517cda92a681346f17797c5931 WatchSource:0}: Error finding container d07d35960ec75ed4d9a13674aea42dc4dec735517cda92a681346f17797c5931: Status 404 returned error can't find the container with id d07d35960ec75ed4d9a13674aea42dc4dec735517cda92a681346f17797c5931 Oct 31 00:21:33.194828 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:33.194744 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" event={"ID":"0af89437-140e-402d-ad56-bda892fda9c8","Type":"ContainerStarted","Data":"d07d35960ec75ed4d9a13674aea42dc4dec735517cda92a681346f17797c5931"} Oct 31 00:21:34.203326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:34.203285 5943 generic.go:356] "Generic (PLEG): container finished" podID="0af89437-140e-402d-ad56-bda892fda9c8" containerID="2519e521b443c008ea9b248cfdc3941a8f08979ebcd1eab45f1e074e03f99b59" exitCode=0 Oct 31 00:21:34.203705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:34.203355 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" event={"ID":"0af89437-140e-402d-ad56-bda892fda9c8","Type":"ContainerDied","Data":"2519e521b443c008ea9b248cfdc3941a8f08979ebcd1eab45f1e074e03f99b59"} Oct 31 00:21:35.354489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.354460 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:35.437976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.437947 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca\") pod \"0af89437-140e-402d-ad56-bda892fda9c8\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " Oct 31 00:21:35.438140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.438099 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d\" (UniqueName: \"kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d\") pod \"0af89437-140e-402d-ad56-bda892fda9c8\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " Oct 31 00:21:35.438232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.438153 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca\") pod \"0af89437-140e-402d-ad56-bda892fda9c8\" (UID: \"0af89437-140e-402d-ad56-bda892fda9c8\") " Oct 31 00:21:35.438609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.438585 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca" pod "0af89437-140e-402d-ad56-bda892fda9c8" (UID: "0af89437-140e-402d-ad56-bda892fda9c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:35.440140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.440121 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca" pod "0af89437-140e-402d-ad56-bda892fda9c8" (UID: "0af89437-140e-402d-ad56-bda892fda9c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:35.440278 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.440252 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d" pod "0af89437-140e-402d-ad56-bda892fda9c8" (UID: "0af89437-140e-402d-ad56-bda892fda9c8"). InnerVolumeSpecName "kube-api-access-62p9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:35.540159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.540120 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62p9d\" (UniqueName: \"kubernetes.io/projected/0af89437-140e-402d-ad56-bda892fda9c8-kube-api-access-62p9d\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:35.540159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.540150 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:35.540159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.540165 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af89437-140e-402d-ad56-bda892fda9c8-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:35.954403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:35.954307 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:36.215563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:36.215469 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" Oct 31 00:21:36.215563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:36.215500 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7" event={"ID":"0af89437-140e-402d-ad56-bda892fda9c8","Type":"ContainerDied","Data":"d07d35960ec75ed4d9a13674aea42dc4dec735517cda92a681346f17797c5931"} Oct 31 00:21:36.215563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:36.215550 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d07d35960ec75ed4d9a13674aea42dc4dec735517cda92a681346f17797c5931" Oct 31 00:21:40.847208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:40.847142 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:43.994378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:43.994343 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.758439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:45.758403 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:49.138804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:49.138713 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:49.361858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:49.361823 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:49.563529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:49.563493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:50.558701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:50.558663 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:52.923435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:52.923397 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:53.394979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:53.394946 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:54.603633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:54.603594 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:21:55.361480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:21:55.361440 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:57.484411 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:21:57.484349 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:22:00.155089 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:00.155053 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:01.199810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:01.199778 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:04.961693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:04.961657 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:09.697264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:09.697230 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:14.516791 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.516759 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp"] Oct 31 00:22:14.517253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.517231 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0af89437-140e-402d-ad56-bda892fda9c8" containerName="revoke-keys" Oct 31 00:22:14.517253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.517246 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af89437-140e-402d-ad56-bda892fda9c8" containerName="revoke-keys" Oct 31 00:22:14.517378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.517367 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af89437-140e-402d-ad56-bda892fda9c8" containerName="revoke-keys" Oct 31 00:22:14.520850 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.520832 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.656528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.656490 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:14.658892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.658862 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659133 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659297 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659470 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659719 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659776 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtrc7\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659817 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.659949 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659859 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.660003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.659944 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.745726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.745687 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp"] Oct 31 00:22:14.760735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760696 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.760937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760745 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.760937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760768 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrc7\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.760937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760792 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.760937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760821 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.760937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760866 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.761255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.760939 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.761255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.761026 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.761255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.761058 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.761549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.761528 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.761790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.761770 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.763247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.762133 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.763247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.762905 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.763247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.763166 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.768253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.765575 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.768253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.765937 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.822305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.822270 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.822638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.822615 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrc7\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7\") pod \"maas-default-gateway-openshift-default-58c6585447-qq4mp\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:14.834103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:14.834062 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:15.274742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.274711 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp"] Oct 31 00:22:15.277648 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:22:15.277625 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c327e5_d646_4e68_9c84_0c94c947d5d3.slice/crio-577869e274a43bd859d5f6c666b85d6dcabbf536ce6981723d099c84a276d2f7 WatchSource:0}: Error finding container 577869e274a43bd859d5f6c666b85d6dcabbf536ce6981723d099c84a276d2f7: Status 404 returned error can't find the container with id 577869e274a43bd859d5f6c666b85d6dcabbf536ce6981723d099c84a276d2f7 Oct 31 00:22:15.279790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.279754 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:22:15.279877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.279824 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:22:15.279877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.279851 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:22:15.432105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.432060 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" event={"ID":"50c327e5-d646-4e68-9c84-0c94c947d5d3","Type":"ContainerStarted","Data":"0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9"} Oct 31 00:22:15.432305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.432140 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" event={"ID":"50c327e5-d646-4e68-9c84-0c94c947d5d3","Type":"ContainerStarted","Data":"577869e274a43bd859d5f6c666b85d6dcabbf536ce6981723d099c84a276d2f7"} Oct 31 00:22:15.554564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.554454 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podStartSLOduration=1.5544356879999999 podStartE2EDuration="1.554435688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:15.552395202 +0000 UTC m=+1115.645685390" watchObservedRunningTime="2026-10-31 00:22:15.554435688 +0000 UTC m=+1115.647725875" Oct 31 00:22:15.841578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:15.836278 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:16.846727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:16.846691 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.847435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:16.847399 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:17.836562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:17.836522 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:17.836886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:17.836844 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:18.840986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:18.840547 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:18.840986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:18.840606 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:19.359226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:19.357549 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:19.564070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:19.564034 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:19.847205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:19.838564 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:19.847205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:19.838625 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:20.849660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:20.849620 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:20.850289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:20.850251 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:21.834809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:21.834773 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:21.834983 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:21.834837 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:22.835624 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:22.835583 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:22:22.836028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:22.835642 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 00:22:23.839399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:23.839369 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:23.839835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:23.839642 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:23.840382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:23.840352 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:22:24.379794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:24.379760 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:25.911166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:25.911124 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:22:27.478131 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:22:27.478090 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:22:29.380551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.380513 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:29.638606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.638519 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm"] Oct 31 00:22:29.642309 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.642288 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.655825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.655804 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.655952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.655841 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.655952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.655916 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtf5\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.656050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.655950 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.656050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.655970 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.656128 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.656101 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.656170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.656128 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.656170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.656159 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.656317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.656297 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.749585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.749544 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm"] Oct 31 00:22:29.757835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.757807 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.757835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.757837 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.757869 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.757902 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.757984 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758033 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758081 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtf5\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758110 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758133 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758274 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758324 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758807 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.758945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758805 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.759001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.758855 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.760728 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.760707 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.761075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.761046 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.834333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.834291 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.835564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.835536 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtf5\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5\") pod \"maas-default-gateway-openshift-default-58c6585447-gltnm\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:29.956426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:29.956346 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:30.386350 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.386323 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm"] Oct 31 00:22:30.388473 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:22:30.388439 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6c8d4d_73ba_4bb3_9b73_251785be67d3.slice/crio-453f60fce6298e97f365272b8c9a432d4c72732919497c902d41a07fa09c51f5 WatchSource:0}: Error finding container 453f60fce6298e97f365272b8c9a432d4c72732919497c902d41a07fa09c51f5: Status 404 returned error can't find the container with id 453f60fce6298e97f365272b8c9a432d4c72732919497c902d41a07fa09c51f5 Oct 31 00:22:30.390835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.390808 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:22:30.390907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.390871 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:22:30.391409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.391374 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:22:30.546234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.546163 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" event={"ID":"1b6c8d4d-73ba-4bb3-9b73-251785be67d3","Type":"ContainerStarted","Data":"386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991"} Oct 31 00:22:30.546426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.546304 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" event={"ID":"1b6c8d4d-73ba-4bb3-9b73-251785be67d3","Type":"ContainerStarted","Data":"453f60fce6298e97f365272b8c9a432d4c72732919497c902d41a07fa09c51f5"} Oct 31 00:22:30.742862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.742740 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podStartSLOduration=1.742726587 podStartE2EDuration="1.742726587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:30.741112435 +0000 UTC m=+1130.834402642" watchObservedRunningTime="2026-10-31 00:22:30.742726587 +0000 UTC m=+1130.836016774" Oct 31 00:22:30.957277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:30.957244 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:31.959207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:31.959163 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:31.959647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:31.959241 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:32.957048 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:32.957013 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:32.957280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:32.957080 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:33.957003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:33.956961 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:33.957448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:33.957028 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:34.520081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:34.520042 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:34.956998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:34.956892 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:34.956998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:34.956966 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:35.957006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:35.956957 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:35.957254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:35.957014 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:36.956770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:36.956735 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:36.956963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:36.956790 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:37.957676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:37.957637 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:22:37.958281 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:37.957721 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 00:22:38.961243 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:38.961211 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:38.961641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:38.961470 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:38.962312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:38.962291 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:22:39.458087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:39.458050 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:44.242572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:44.242539 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:44.392454 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:44.392413 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:49.167447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:49.167408 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:53.970387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:53.970356 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:57.476140 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:22:57.476102 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:22:58.770634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:22:58.770591 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:03.555138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:03.555095 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:04.258830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.258792 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r"] Oct 31 00:23:04.262543 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.262519 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.353110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.353072 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.353110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.353111 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.353356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.353154 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.353356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.353226 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.353423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.353352 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.353476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.353455 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2zs\" (UniqueName: \"kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.356541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.356518 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:04.364253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.364228 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"distinct-w1-869cca-kserve-self-signed-certs\"" Oct 31 00:23:04.366046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.366027 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r"] Oct 31 00:23:04.454160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454117 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454161 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454216 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454238 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454324 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454400 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2zs\" (UniqueName: \"kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454630 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454665 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.454820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.454798 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.457310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.457280 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.457549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.457533 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.516249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.516135 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2zs\" (UniqueName: \"kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs\") pod \"distinct-w1-869cca-kserve-54cdd69cf4-jff7r\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.574955 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.574909 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:04.984920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:04.984890 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r"] Oct 31 00:23:04.987128 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:23:04.987101 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67438e1_87b4_4676_ba1b_964c3e3bcedb.slice/crio-9ca5859f7e0fbef75dd48ed450385b89fa68b3ae40807054892dab9b2cc846e9 WatchSource:0}: Error finding container 9ca5859f7e0fbef75dd48ed450385b89fa68b3ae40807054892dab9b2cc846e9: Status 404 returned error can't find the container with id 9ca5859f7e0fbef75dd48ed450385b89fa68b3ae40807054892dab9b2cc846e9 Oct 31 00:23:05.758793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:05.758756 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" event={"ID":"a67438e1-87b4-4676-ba1b-964c3e3bcedb","Type":"ContainerStarted","Data":"3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9"} Oct 31 00:23:05.759348 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:05.758816 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" event={"ID":"a67438e1-87b4-4676-ba1b-964c3e3bcedb","Type":"ContainerStarted","Data":"9ca5859f7e0fbef75dd48ed450385b89fa68b3ae40807054892dab9b2cc846e9"} Oct 31 00:23:05.875544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:05.875489 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podStartSLOduration=1.8754752369999999 podStartE2EDuration="1.875475237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:05.874080224 +0000 UTC m=+1165.967370441" watchObservedRunningTime="2026-10-31 00:23:05.875475237 +0000 UTC m=+1165.968765423" Oct 31 00:23:06.575467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:06.575431 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:07.626251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.626216 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w"] Oct 31 00:23:07.630049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.630029 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.693255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.693219 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.693426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.693322 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.693426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.693357 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.693542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.693444 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.693542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.693484 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.693635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.693570 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6whd\" (UniqueName: \"kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.710401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.710373 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"default-dockercfg-nb88j\"" Oct 31 00:23:07.733508 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.733478 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"distinct-w2-e1ca71-kserve-self-signed-certs\"" Oct 31 00:23:07.733709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.733594 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"kube-root-ca.crt\"" Oct 31 00:23:07.733709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.733608 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"openshift-service-ca.crt\"" Oct 31 00:23:07.735498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.735480 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w"] Oct 31 00:23:07.746689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.746645 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:07.794529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.794495 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.794742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.794547 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.794742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.794629 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6whd\" (UniqueName: \"kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.794742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.794711 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.794917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.794762 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.794917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.794797 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.795089 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.795062 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.795157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.795089 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.795537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.795494 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.798100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.798074 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.798396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.798377 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.856636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.856603 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6whd\" (UniqueName: \"kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd\") pod \"distinct-w2-e1ca71-kserve-647d5f889-4z26w\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:07.943416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:07.943327 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:08.336115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:08.336087 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w"] Oct 31 00:23:08.338594 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:23:08.338561 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67acf536_3524_471c_babf_7a1af4cc4a4e.slice/crio-61be546ab7ba44cbf15cb8ff0a28b5805f3435c4b357387a4163e6dff20e4aa5 WatchSource:0}: Error finding container 61be546ab7ba44cbf15cb8ff0a28b5805f3435c4b357387a4163e6dff20e4aa5: Status 404 returned error can't find the container with id 61be546ab7ba44cbf15cb8ff0a28b5805f3435c4b357387a4163e6dff20e4aa5 Oct 31 00:23:08.353349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:08.353320 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:08.777389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:08.777351 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" event={"ID":"67acf536-3524-471c-babf-7a1af4cc4a4e","Type":"ContainerStarted","Data":"56d474696e033f6250356c7f3e86c4ae8b9c3c2fec3e24edc207b40c8c253101"} Oct 31 00:23:08.777389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:08.777405 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" event={"ID":"67acf536-3524-471c-babf-7a1af4cc4a4e","Type":"ContainerStarted","Data":"61be546ab7ba44cbf15cb8ff0a28b5805f3435c4b357387a4163e6dff20e4aa5"} Oct 31 00:23:08.893745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:08.893689 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podStartSLOduration=1.8936751649999999 podStartE2EDuration="1.893675165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:08.893169929 +0000 UTC m=+1168.986460134" watchObservedRunningTime="2026-10-31 00:23:08.893675165 +0000 UTC m=+1168.986965435" Oct 31 00:23:08.943915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:08.943880 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:13.247980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:13.247942 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:14.576145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:14.576109 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:14.588708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:14.588682 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:14.823324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:14.823292 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:23:15.037713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:15.037681 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:23:17.943461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:17.943427 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:17.957650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:17.957622 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:18.179580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:18.179540 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:18.845694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:18.845616 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:23:22.960220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:22.960161 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:27.476162 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:23:27.476025 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:23:27.737813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:27.737730 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:29.611144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:29.611099 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:31.124640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.124602 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb"] Oct 31 00:23:31.128546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.128524 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169338 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169387 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169507 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169639 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169560 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169639 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169599 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169674 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169704 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169778 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsrq\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.169918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.169888 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.180633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.180605 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e1ca71-openshift-default-dockercfg-sh7lb\"" Oct 31 00:23:31.234943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.234896 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb"] Oct 31 00:23:31.271634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271592 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.271836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271665 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.271836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271711 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.271836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271732 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271862 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271918 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.271974 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsrq\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272038 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272057 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272087 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272158 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272280 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272161 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.272753 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.272734 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.274197 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.274150 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.274563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.274543 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.333088 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.333044 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.333289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.333243 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsrq\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.443756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.443668 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:31.871849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.871819 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb"] Oct 31 00:23:31.873222 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:23:31.873193 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc253cb_40fc_4786_9c1b_c1dd555ef4b3.slice/crio-c84bc0d270c676515d42ac6944803fb248ef0402f6405bba3ea920a87eb8887c WatchSource:0}: Error finding container c84bc0d270c676515d42ac6944803fb248ef0402f6405bba3ea920a87eb8887c: Status 404 returned error can't find the container with id c84bc0d270c676515d42ac6944803fb248ef0402f6405bba3ea920a87eb8887c Oct 31 00:23:31.875898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.875869 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:23:31.875975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.875935 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:23:31.875975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.875960 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:23:31.909894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:31.909854 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" event={"ID":"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3","Type":"ContainerStarted","Data":"c84bc0d270c676515d42ac6944803fb248ef0402f6405bba3ea920a87eb8887c"} Oct 31 00:23:32.570192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:32.570134 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:32.920546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:32.920448 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" event={"ID":"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3","Type":"ContainerStarted","Data":"01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7"} Oct 31 00:23:33.147206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:33.146509 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podStartSLOduration=2.146489453 podStartE2EDuration="2.146489453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:33.090577303 +0000 UTC m=+1193.183867505" watchObservedRunningTime="2026-10-31 00:23:33.146489453 +0000 UTC m=+1193.239779641" Oct 31 00:23:33.444606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:33.444570 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:33.450666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:33.450632 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:23:33.450912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:33.450873 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 00:23:34.445204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:34.444625 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:23:34.445204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:34.444694 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 00:23:35.448206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:35.445082 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:23:35.448206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:35.445152 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 00:23:35.518206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:35.512278 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:35.597697 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:35.597307 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:36.145205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:36.144286 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:36.445663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:36.445563 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:23:36.445939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:36.445895 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 00:23:37.453665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:37.453622 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:23:37.454233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:37.453692 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 00:23:37.877232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:37.877198 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:38.444837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:38.444798 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:23:38.445061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:38.444858 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 00:23:39.449709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:39.449679 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:39.450133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:39.450114 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:39.450981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:39.450964 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 00:23:40.192425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:40.192381 5943 scope.go:122] "RemoveContainer" containerID="126026312cf2de05ce5b85b08d6fe5f912c69a29963e27f93062998a3340ca26" Oct 31 00:23:40.202066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:40.202046 5943 scope.go:122] "RemoveContainer" containerID="8b57cfaeaa1da63aed065e0d105b07e61aee9b6c9cab94ce0d52872773052417" Oct 31 00:23:42.748328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:42.748286 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:44.695974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:44.695935 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:44.917037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:44.916994 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:23:47.759141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:47.759104 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:52.634781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:52.634745 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:57.455509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:23:57.455471 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:57.478968 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:23:57.478932 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:24:01.468958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.468918 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2"] Oct 31 00:24:01.472766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.472746 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.563037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.563003 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2"] Oct 31 00:24:01.569929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.569884 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570022 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570132 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570243 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570271 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570353 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hw74\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570388 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570426 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.570595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.570504 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671136 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hw74\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671219 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671264 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671301 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671378 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671423 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671520 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671610 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671771 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671637 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671771 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671709 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671771 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671726 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.671965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671933 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.672037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.671961 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.672273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.672227 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.673962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.673937 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.674207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.674163 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.732478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.732408 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.732702 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.732629 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hw74\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:01.786555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:01.786520 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:02.218548 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:02.218521 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2"] Oct 31 00:24:02.221261 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:24:02.221235 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9169c6_9490_4291_9369_1fd3ac5dafbe.slice/crio-7e61bb314c2f946152c39a16d177c15942033a1cef7568e4502f90deef5b530d WatchSource:0}: Error finding container 7e61bb314c2f946152c39a16d177c15942033a1cef7568e4502f90deef5b530d: Status 404 returned error can't find the container with id 7e61bb314c2f946152c39a16d177c15942033a1cef7568e4502f90deef5b530d Oct 31 00:24:02.223597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:02.223565 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:24:02.223663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:02.223634 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:24:02.223700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:02.223663 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:24:02.256523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:02.256476 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:03.212616 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:03.212583 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" event={"ID":"4c9169c6-9490-4291-9369-1fd3ac5dafbe","Type":"ContainerStarted","Data":"8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa"} Oct 31 00:24:03.213032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:03.212637 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" event={"ID":"4c9169c6-9490-4291-9369-1fd3ac5dafbe","Type":"ContainerStarted","Data":"7e61bb314c2f946152c39a16d177c15942033a1cef7568e4502f90deef5b530d"} Oct 31 00:24:03.384006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:03.383952 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podStartSLOduration=2.383935494 podStartE2EDuration="2.383935494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:03.381679956 +0000 UTC m=+1223.474970163" watchObservedRunningTime="2026-10-31 00:24:03.383935494 +0000 UTC m=+1223.477225680" Oct 31 00:24:03.787715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:03.787681 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:03.789388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:03.789357 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:24:03.789541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:03.789418 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 00:24:04.787089 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:04.787057 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:24:04.787531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:04.787112 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 00:24:05.786806 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:05.786768 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:24:05.787016 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:05.786837 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 00:24:05.874215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:05.874147 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:06.022171 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:06.022136 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:06.787354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:06.787314 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:24:06.787552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:06.787385 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 00:24:07.095115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:07.094994 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:07.787828 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:07.787793 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:24:07.788047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:07.787853 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 00:24:08.057508 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:08.051822 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:08.787160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:08.787127 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:24:08.787677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:08.787206 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 00:24:09.875236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:09.875197 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:09.876213 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:09.876170 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:09.877430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:09.877412 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:24:11.945587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:11.945551 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:16.694751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:16.694714 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:16.974454 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:16.974370 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:24:21.623051 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:21.623002 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:21.789628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:21.789586 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:26.303096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:26.303060 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:27.477968 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:24:27.477931 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:24:31.042286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:31.042247 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:35.893536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:35.893495 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:36.951286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:36.951247 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:37.148613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:37.148573 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:37.306740 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:37.306703 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:40.502025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:40.501991 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:40.762713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:40.762662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:42.655739 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:42.655702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:45.053152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:45.053107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:45.553318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:45.553277 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:48.835766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:48.835675 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:24:50.474615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:50.474559 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:55.451527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:55.451489 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:56.433416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:56.433374 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:56.630468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:24:56.630430 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:57.477096 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:24:57.477060 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:25:00.235944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:00.235904 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:04.989657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:04.989626 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:06.771377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:06.771343 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:09.958159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:09.958112 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:14.779207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:14.778854 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:17.325024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:17.324978 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:18.775612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:18.775572 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:19.657649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:19.657602 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:22.844081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.844049 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds"] Oct 31 00:25:22.848393 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.848370 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.900570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.900539 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-869cca-openshift-default-dockercfg-49wg8\"" Oct 31 00:25:22.909195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909141 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909203 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909318 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vdc\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909351 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909373 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909511 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909562 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909607 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.909675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.909662 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:22.953456 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:22.953405 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds"] Oct 31 00:25:23.010903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.010850 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vdc\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.010916 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.010947 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011012 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011069 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011250 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011371 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011434 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011476 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011486 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011546 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011691 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.011796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.011777 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.012022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.012004 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.013538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.013513 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.014215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.014194 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.072683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.072656 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vdc\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.072831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.072659 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.162486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.162396 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:23.595236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.595206 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds"] Oct 31 00:25:23.597152 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:25:23.597118 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9624f662_6a04_4586_b941_ba89204173ba.slice/crio-e2bc9da3b261ed997f4287bce4c19c5f00b07f9a2f9b21e4d40161d597d9145c WatchSource:0}: Error finding container e2bc9da3b261ed997f4287bce4c19c5f00b07f9a2f9b21e4d40161d597d9145c: Status 404 returned error can't find the container with id e2bc9da3b261ed997f4287bce4c19c5f00b07f9a2f9b21e4d40161d597d9145c Oct 31 00:25:23.599599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.599566 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:25:23.599668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.599642 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:25:23.599705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.599667 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:25:23.747792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.747751 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" event={"ID":"9624f662-6a04-4586-b941-ba89204173ba","Type":"ContainerStarted","Data":"aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4"} Oct 31 00:25:23.747987 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.747823 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" event={"ID":"9624f662-6a04-4586-b941-ba89204173ba","Type":"ContainerStarted","Data":"e2bc9da3b261ed997f4287bce4c19c5f00b07f9a2f9b21e4d40161d597d9145c"} Oct 31 00:25:23.870542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:23.870429 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podStartSLOduration=1.8704144870000001 podStartE2EDuration="1.870414487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:23.867052177 +0000 UTC m=+1303.960342389" watchObservedRunningTime="2026-10-31 00:25:23.870414487 +0000 UTC m=+1303.963704675" Oct 31 00:25:24.163451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:24.163364 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:24.165039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:24.165010 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:24.165150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:24.165066 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:24.474467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:24.474371 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:25.163796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:25.163758 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:25.164319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:25.163851 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:26.163107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:26.163014 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:26.163107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:26.163072 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:27.163751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:27.163707 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:27.164198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:27.163769 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:27.483890 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:25:27.483803 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:25:28.163115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:28.163074 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:28.163349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:28.163157 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:29.163884 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:29.163838 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:29.164332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:29.163937 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:29.386252 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:29.386218 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:30.163725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:30.163693 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:25:30.163929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:30.163754 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 00:25:31.167703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:31.167671 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:31.168162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:31.167977 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:31.168798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:31.168782 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:25:34.112680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:34.112637 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:25:34.155818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:34.155781 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:38.964730 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:38.964677 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:39.419110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:39.419049 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:43.839064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:43.839030 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:44.721650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.721613 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr"] Oct 31 00:25:44.725916 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.725886 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.778123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.778096 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:25:44.797796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.797764 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.797966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.797801 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.797966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.797950 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.827882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.827851 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:44.830129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.830107 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr"] Oct 31 00:25:44.831108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.831093 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:25:44.878291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.878259 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:25:44.899475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.899441 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.899690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.899561 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.899690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.899583 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.900173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.900146 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.902150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.902128 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:44.960565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:44.960532 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s\") pod \"maas-api-revoke-sub-71460944-qdtgr\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:45.040067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:45.040031 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:45.395915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:45.395813 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:45.419613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:45.419588 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr"] Oct 31 00:25:45.422697 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:25:45.422663 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57616b1b_74a1_45da_87e0_55e6007ad9b4.slice/crio-4dc57251fe710428d9fe6dea5974354ba61520d2f556f0341e4c654e78bb02e3 WatchSource:0}: Error finding container 4dc57251fe710428d9fe6dea5974354ba61520d2f556f0341e4c654e78bb02e3: Status 404 returned error can't find the container with id 4dc57251fe710428d9fe6dea5974354ba61520d2f556f0341e4c654e78bb02e3 Oct 31 00:25:45.890868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:45.890829 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" event={"ID":"57616b1b-74a1-45da-87e0-55e6007ad9b4","Type":"ContainerStarted","Data":"f4b5b53a05a228f1983eb79326311f043f76e354d7ea1b78358fa96caaa1e8c6"} Oct 31 00:25:45.891297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:45.890896 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" event={"ID":"57616b1b-74a1-45da-87e0-55e6007ad9b4","Type":"ContainerStarted","Data":"4dc57251fe710428d9fe6dea5974354ba61520d2f556f0341e4c654e78bb02e3"} Oct 31 00:25:46.028769 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:46.028707 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" podStartSLOduration=2.028693325 podStartE2EDuration="2.028693325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:46.026951 +0000 UTC m=+1326.120241187" watchObservedRunningTime="2026-10-31 00:25:46.028693325 +0000 UTC m=+1326.121983511" Oct 31 00:25:46.898034 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:46.897996 5943 generic.go:356] "Generic (PLEG): container finished" podID="57616b1b-74a1-45da-87e0-55e6007ad9b4" containerID="f4b5b53a05a228f1983eb79326311f043f76e354d7ea1b78358fa96caaa1e8c6" exitCode=0 Oct 31 00:25:46.898499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:46.898090 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" event={"ID":"57616b1b-74a1-45da-87e0-55e6007ad9b4","Type":"ContainerDied","Data":"f4b5b53a05a228f1983eb79326311f043f76e354d7ea1b78358fa96caaa1e8c6"} Oct 31 00:25:48.080681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.080654 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:48.136850 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.136814 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca\") pod \"57616b1b-74a1-45da-87e0-55e6007ad9b4\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " Oct 31 00:25:48.137032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.136974 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca\") pod \"57616b1b-74a1-45da-87e0-55e6007ad9b4\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " Oct 31 00:25:48.137084 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.137037 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s\") pod \"57616b1b-74a1-45da-87e0-55e6007ad9b4\" (UID: \"57616b1b-74a1-45da-87e0-55e6007ad9b4\") " Oct 31 00:25:48.137436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.137409 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca" pod "57616b1b-74a1-45da-87e0-55e6007ad9b4" (UID: "57616b1b-74a1-45da-87e0-55e6007ad9b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:48.139286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.139260 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca" pod "57616b1b-74a1-45da-87e0-55e6007ad9b4" (UID: "57616b1b-74a1-45da-87e0-55e6007ad9b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:48.139473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.139449 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s" pod "57616b1b-74a1-45da-87e0-55e6007ad9b4" (UID: "57616b1b-74a1-45da-87e0-55e6007ad9b4"). InnerVolumeSpecName "kube-api-access-k769s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:48.238783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.238701 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:48.238783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.238734 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k769s\" (UniqueName: \"kubernetes.io/projected/57616b1b-74a1-45da-87e0-55e6007ad9b4-kube-api-access-k769s\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:48.238783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.238744 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57616b1b-74a1-45da-87e0-55e6007ad9b4-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:48.679528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.679493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:48.910205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.910162 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" Oct 31 00:25:48.910403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.910207 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr" event={"ID":"57616b1b-74a1-45da-87e0-55e6007ad9b4","Type":"ContainerDied","Data":"4dc57251fe710428d9fe6dea5974354ba61520d2f556f0341e4c654e78bb02e3"} Oct 31 00:25:48.910403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:48.910248 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc57251fe710428d9fe6dea5974354ba61520d2f556f0341e4c654e78bb02e3" Oct 31 00:25:53.466953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:53.466875 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:56.593133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:56.593089 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:56.739507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:56.739351 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:56.897815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:56.897719 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:57.477850 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:25:57.477814 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:25:58.366917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:25:58.366883 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:02.309747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.309708 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5"] Oct 31 00:26:02.310563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.310542 5943 cpu_manager.go:414] "containerMap: removing container" podUID="57616b1b-74a1-45da-87e0-55e6007ad9b4" containerName="revoke-keys" Oct 31 00:26:02.310563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.310563 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57616b1b-74a1-45da-87e0-55e6007ad9b4" containerName="revoke-keys" Oct 31 00:26:02.310728 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.310661 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="57616b1b-74a1-45da-87e0-55e6007ad9b4" containerName="revoke-keys" Oct 31 00:26:02.314342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.314326 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.387001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.386967 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl"] Oct 31 00:26:02.390758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.390736 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.411405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411367 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411417 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411503 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411563 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411587 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zx7s\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411650 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411680 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.411766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411694 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.412012 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.411847 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.423359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.423328 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:02.443933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.443900 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"distinct-w0-10a7aa-kserve-self-signed-certs\"" Oct 31 00:26:02.450528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.450496 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl"] Oct 31 00:26:02.513044 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513005 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513085 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513137 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfpb6\" (UniqueName: \"kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.513228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513167 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513594 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513608 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513643 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.513849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513685 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513712 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zx7s\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513645 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.513849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513804 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.513849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513845 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513881 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513933 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.513957 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.514012 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.514045 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.514661 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.514665 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.515737 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.520580 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.524140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.522090 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.534213 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.529145 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5"] Oct 31 00:26:02.574342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.574249 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.574342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.574273 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zx7s\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.615301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615259 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615355 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615398 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615464 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615496 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615673 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615764 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfpb6\" (UniqueName: \"kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.615796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.615772 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.616047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.616025 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.618247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.618222 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.618365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.618225 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.628941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.628912 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:02.677672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.677628 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfpb6\" (UniqueName: \"kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6\") pod \"distinct-w0-10a7aa-kserve-d879cd794-sswcl\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:02.734270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:02.731486 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:03.083623 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:03.083592 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5"] Oct 31 00:26:03.090024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:03.089994 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:03.090108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:03.090055 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:03.090108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:03.090084 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:03.181906 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:03.181813 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl"] Oct 31 00:26:03.185770 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:03.185731 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f683ac1_9ac3_4174_b3c9_9d8a8a1396e5.slice/crio-4de89e99e7a34655eba41a0e96657733197958eb01581ab08068d67041480efe WatchSource:0}: Error finding container 4de89e99e7a34655eba41a0e96657733197958eb01581ab08068d67041480efe: Status 404 returned error can't find the container with id 4de89e99e7a34655eba41a0e96657733197958eb01581ab08068d67041480efe Oct 31 00:26:03.248507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:03.248416 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:04.000247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.000197 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" event={"ID":"39b6a45e-1e54-4170-81ed-65f7364ff9a6","Type":"ContainerStarted","Data":"a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d"} Oct 31 00:26:04.000732 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.000258 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" event={"ID":"39b6a45e-1e54-4170-81ed-65f7364ff9a6","Type":"ContainerStarted","Data":"5312d3eafa5aa3549874c10b0805e86b78c6e667297ec50aca33b7d1f1c55183"} Oct 31 00:26:04.001868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.001843 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" event={"ID":"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5","Type":"ContainerStarted","Data":"ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843"} Oct 31 00:26:04.002023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.001875 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" event={"ID":"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5","Type":"ContainerStarted","Data":"4de89e99e7a34655eba41a0e96657733197958eb01581ab08068d67041480efe"} Oct 31 00:26:04.171870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.171813 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podStartSLOduration=2.171799883 podStartE2EDuration="2.171799883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:04.171071777 +0000 UTC m=+1344.264361990" watchObservedRunningTime="2026-10-31 00:26:04.171799883 +0000 UTC m=+1344.265090060" Oct 31 00:26:04.288230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.288157 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podStartSLOduration=2.288139216 podStartE2EDuration="2.288139216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:04.286914132 +0000 UTC m=+1344.380204319" watchObservedRunningTime="2026-10-31 00:26:04.288139216 +0000 UTC m=+1344.381429420" Oct 31 00:26:04.640267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.636578 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:04.653380 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.653323 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:26:04.653575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.653424 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:26:04.750205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:04.742848 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:05.272250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.272214 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r"] Oct 31 00:26:05.275903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.275883 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.349207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.328373 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:05.349455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.328824 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:05.349583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.343395 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:26:05.349667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.343521 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:26:05.376378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.376341 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.376560 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.376399 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g962k\" (UniqueName: \"kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.376668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.376640 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.380278 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.380256 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r"] Oct 31 00:26:05.477376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.477343 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.477545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.477468 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.477545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.477526 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g962k\" (UniqueName: \"kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.478374 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.478323 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.479979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.479959 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.555635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.555567 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g962k\" (UniqueName: \"kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k\") pod \"maas-api-revoke-sub-a7cf7775-krb6r\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.590693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.590666 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:05.655549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.654639 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:26:05.655549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.654705 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:26:05.992501 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:05.992468 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r"] Oct 31 00:26:05.993976 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:05.993944 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5459a02_388c_4c60_8376_f937b05a68de.slice/crio-b86d03afe66fca8ab9a849ee6c72d0f2d440900c73d87660c07e2c2ee46ff642 WatchSource:0}: Error finding container b86d03afe66fca8ab9a849ee6c72d0f2d440900c73d87660c07e2c2ee46ff642: Status 404 returned error can't find the container with id b86d03afe66fca8ab9a849ee6c72d0f2d440900c73d87660c07e2c2ee46ff642 Oct 31 00:26:06.016107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:06.016074 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" event={"ID":"b5459a02-388c-4c60-8376-f937b05a68de","Type":"ContainerStarted","Data":"b86d03afe66fca8ab9a849ee6c72d0f2d440900c73d87660c07e2c2ee46ff642"} Oct 31 00:26:06.638264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:06.629806 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:26:06.638264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:06.629867 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:26:07.024085 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:07.024045 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" event={"ID":"b5459a02-388c-4c60-8376-f937b05a68de","Type":"ContainerStarted","Data":"cd9285380b2a792e7e2a7805b52e13670c4805d5439bb56638be4756bbc77d1d"} Oct 31 00:26:07.637611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:07.637574 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:26:07.637877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:07.637848 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:26:08.055403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:08.055370 5943 generic.go:356] "Generic (PLEG): container finished" podID="b5459a02-388c-4c60-8376-f937b05a68de" containerID="cd9285380b2a792e7e2a7805b52e13670c4805d5439bb56638be4756bbc77d1d" exitCode=0 Oct 31 00:26:08.059103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:08.059073 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" event={"ID":"b5459a02-388c-4c60-8376-f937b05a68de","Type":"ContainerDied","Data":"cd9285380b2a792e7e2a7805b52e13670c4805d5439bb56638be4756bbc77d1d"} Oct 31 00:26:08.194214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:08.194151 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:08.646246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:08.644700 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:26:08.646246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:08.644779 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:26:09.221688 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.221649 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:09.226640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.226612 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca\") pod \"b5459a02-388c-4c60-8376-f937b05a68de\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " Oct 31 00:26:09.226782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.226764 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca\") pod \"b5459a02-388c-4c60-8376-f937b05a68de\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " Oct 31 00:26:09.226903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.226887 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k\" (UniqueName: \"kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k\") pod \"b5459a02-388c-4c60-8376-f937b05a68de\" (UID: \"b5459a02-388c-4c60-8376-f937b05a68de\") " Oct 31 00:26:09.227137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.227110 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca" pod "b5459a02-388c-4c60-8376-f937b05a68de" (UID: "b5459a02-388c-4c60-8376-f937b05a68de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:09.227461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.227439 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:09.228912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.228889 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca" pod "b5459a02-388c-4c60-8376-f937b05a68de" (UID: "b5459a02-388c-4c60-8376-f937b05a68de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:09.229045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.229029 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k" pod "b5459a02-388c-4c60-8376-f937b05a68de" (UID: "b5459a02-388c-4c60-8376-f937b05a68de"). InnerVolumeSpecName "kube-api-access-g962k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:09.328910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.328826 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g962k\" (UniqueName: \"kubernetes.io/projected/b5459a02-388c-4c60-8376-f937b05a68de-kube-api-access-g962k\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:09.328910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.328855 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5459a02-388c-4c60-8376-f937b05a68de-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:09.630061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.629969 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:26:09.630061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:09.630034 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:26:10.069781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.069748 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" event={"ID":"b5459a02-388c-4c60-8376-f937b05a68de","Type":"ContainerDied","Data":"b86d03afe66fca8ab9a849ee6c72d0f2d440900c73d87660c07e2c2ee46ff642"} Oct 31 00:26:10.069985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.069785 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r" Oct 31 00:26:10.069985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.069802 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b86d03afe66fca8ab9a849ee6c72d0f2d440900c73d87660c07e2c2ee46ff642" Oct 31 00:26:10.483877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.483758 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:10.634292 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.634255 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:10.634604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.634583 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:10.634913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.634895 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:10.635640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:10.635624 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:26:12.733798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:12.733766 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:12.746697 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:12.746675 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:12.948653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:12.948617 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:13.101630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:13.101590 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:26:16.389717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:16.389684 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:26:17.948204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:17.935891 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:18.454164 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.454118 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:18.573356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.573304 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:18.737262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.736287 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 00:26:18.795577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.793297 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 00:26:18.823410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.823365 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 00:26:18.850250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.850209 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 00:26:18.851749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.851718 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:18.946382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:18.946273 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.101215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.099294 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.106214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.105284 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.145978 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.143477 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 00:26:19.457015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.456896 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.458497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.458474 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:26:19.479417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.479388 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:26:19.574794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.574738 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.576046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.576024 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:26:19.777461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.776314 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.777461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.776801 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:26:19.846204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.844341 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 00:26:19.846204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.844767 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:26:19.858502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.858463 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.859091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.859068 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:26:19.859262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.859221 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.864199 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.860952 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:26:19.864440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.864423 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:26:19.889473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.887313 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:26:19.913207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:19.910125 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:26:20.131127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:20.123504 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:20.226218 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:20.216732 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:26:20.359206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:20.357039 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:26:22.851307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.851263 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz"] Oct 31 00:26:22.852009 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.851986 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b5459a02-388c-4c60-8376-f937b05a68de" containerName="revoke-keys" Oct 31 00:26:22.852106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.852010 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5459a02-388c-4c60-8376-f937b05a68de" containerName="revoke-keys" Oct 31 00:26:22.852157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.852128 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5459a02-388c-4c60-8376-f937b05a68de" containerName="revoke-keys" Oct 31 00:26:22.856719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.856697 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936328 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936375 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6pb\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936446 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936491 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936624 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936668 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936749 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936779 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:22.936829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:22.936816 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038471 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038519 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038556 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038588 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038626 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038682 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038772 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6pb\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038826 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.038986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.038885 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.039136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.039009 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.039301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.039275 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.039421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.039352 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.039421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.039408 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.039658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.039637 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.041108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.041086 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.041353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.041337 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.075963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.075932 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz"] Oct 31 00:26:23.100521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.100452 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.100705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.100525 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6pb\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.171319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.171221 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:23.384984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.384944 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:23.607148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.607123 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz"] Oct 31 00:26:23.608844 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:23.608811 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0c55fa_7995_49b8_9fd0_81c664d7b8a7.slice/crio-61c1c4e4f2fe341b9904fcfa6cf299ad873a0a4e33189f394d11b006f9c38d77 WatchSource:0}: Error finding container 61c1c4e4f2fe341b9904fcfa6cf299ad873a0a4e33189f394d11b006f9c38d77: Status 404 returned error can't find the container with id 61c1c4e4f2fe341b9904fcfa6cf299ad873a0a4e33189f394d11b006f9c38d77 Oct 31 00:26:23.611532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.611496 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:23.611638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.611579 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:23.611638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:23.611624 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:24.273388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:24.273357 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" event={"ID":"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7","Type":"ContainerStarted","Data":"58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9"} Oct 31 00:26:24.273794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:24.273414 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" event={"ID":"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7","Type":"ContainerStarted","Data":"61c1c4e4f2fe341b9904fcfa6cf299ad873a0a4e33189f394d11b006f9c38d77"} Oct 31 00:26:25.171827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:25.171793 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:25.173526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:25.173491 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:26:25.173671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:25.173556 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 00:26:26.172575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:26.172481 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:26:26.172575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:26.172555 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 00:26:27.171892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:27.171857 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:26:27.172080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:27.171920 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 00:26:27.486376 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:26:27.486279 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:26:28.172596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:28.172561 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:26:28.172790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:28.172633 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 00:26:28.292964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:28.292924 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:29.172036 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:29.171998 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:26:29.172467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:29.172060 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 00:26:30.171955 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:30.171919 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:26:30.172148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:30.171978 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 00:26:31.176229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:31.176197 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:31.176664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:31.176475 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:31.177116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:31.177093 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:26:31.349223 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:31.349155 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podStartSLOduration=9.349136687 podStartE2EDuration="9.349136687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:24.455035396 +0000 UTC m=+1364.548325586" watchObservedRunningTime="2026-10-31 00:26:31.349136687 +0000 UTC m=+1371.442426874" Oct 31 00:26:32.145744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.145709 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f"] Oct 31 00:26:32.149353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.149336 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.253551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.253523 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:26:32.253973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.253560 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:32.253973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.253618 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:32.253973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.253639 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:26:32.256075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.256053 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f"] Oct 31 00:26:32.262960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.262938 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54lx\" (UniqueName: \"kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.263068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.262974 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.263165 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.263146 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.364407 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.364365 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54lx\" (UniqueName: \"kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.364657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.364423 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.364657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.364548 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.365340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.365319 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.367136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.367093 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.425873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.425802 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54lx\" (UniqueName: \"kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx\") pod \"maas-api-revoke-sub-5d547547-smk9f\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.466538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.466500 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:32.625255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.625220 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs"] Oct 31 00:26:32.629332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.629303 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.716592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.716510 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs"] Oct 31 00:26:32.770696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.770658 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.770877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.770725 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.770877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.770803 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.770877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.770858 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.771000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.770937 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.771000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.770987 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmq6\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.771098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.771079 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.771136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.771115 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.771210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.771138 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.853985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.853960 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f"] Oct 31 00:26:32.856146 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:32.856115 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1036eeff_51ee_40e4_ae45_fb725bb931dd.slice/crio-9a39a977571909f6d75b80a246135e49361cb4976a773b1b339284d301a7bb16 WatchSource:0}: Error finding container 9a39a977571909f6d75b80a246135e49361cb4976a773b1b339284d301a7bb16: Status 404 returned error can't find the container with id 9a39a977571909f6d75b80a246135e49361cb4976a773b1b339284d301a7bb16 Oct 31 00:26:32.872923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.872893 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.872995 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873034 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873105 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873142 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873233 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmq6\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873299 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873332 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873336 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873401 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873515 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873419 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873703 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873716 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.873856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.873822 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.875789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.875767 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.875900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.875883 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.934023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.933996 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.934273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.934254 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmq6\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:32.943097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:32.943077 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:33.241490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.241456 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:33.331669 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.331619 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" event={"ID":"1036eeff-51ee-40e4-ae45-fb725bb931dd","Type":"ContainerStarted","Data":"c7fe53ffb31b53841e977e91e111b4267b0262a56f3e2bd49b9859314cf15f0b"} Oct 31 00:26:33.332290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.331706 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" event={"ID":"1036eeff-51ee-40e4-ae45-fb725bb931dd","Type":"ContainerStarted","Data":"9a39a977571909f6d75b80a246135e49361cb4976a773b1b339284d301a7bb16"} Oct 31 00:26:33.374034 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.374004 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs"] Oct 31 00:26:33.377708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:33.377672 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212539d4_2703_4468_932e_66b898149beb.slice/crio-2a77dccf42088a0366e9bfa05bb101380a8f0d41393de6ec7b06170529824d09 WatchSource:0}: Error finding container 2a77dccf42088a0366e9bfa05bb101380a8f0d41393de6ec7b06170529824d09: Status 404 returned error can't find the container with id 2a77dccf42088a0366e9bfa05bb101380a8f0d41393de6ec7b06170529824d09 Oct 31 00:26:33.380432 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.380394 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:33.380529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.380478 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:33.380529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.380520 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:33.447534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:33.447483 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" podStartSLOduration=1.4474667 podStartE2EDuration="1.4474667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:33.446999504 +0000 UTC m=+1373.540289719" watchObservedRunningTime="2026-10-31 00:26:33.4474667 +0000 UTC m=+1373.540756914" Oct 31 00:26:34.361634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.361013 5943 generic.go:356] "Generic (PLEG): container finished" podID="1036eeff-51ee-40e4-ae45-fb725bb931dd" containerID="c7fe53ffb31b53841e977e91e111b4267b0262a56f3e2bd49b9859314cf15f0b" exitCode=0 Oct 31 00:26:34.361634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.361122 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" event={"ID":"1036eeff-51ee-40e4-ae45-fb725bb931dd","Type":"ContainerDied","Data":"c7fe53ffb31b53841e977e91e111b4267b0262a56f3e2bd49b9859314cf15f0b"} Oct 31 00:26:34.363850 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.363802 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" event={"ID":"212539d4-2703-4468-932e-66b898149beb","Type":"ContainerStarted","Data":"76e287a6dc546d15fd6a8d98ddf4f8df6a78a2c542cec55eff5b417ce896a05b"} Oct 31 00:26:34.364000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.363880 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" event={"ID":"212539d4-2703-4468-932e-66b898149beb","Type":"ContainerStarted","Data":"2a77dccf42088a0366e9bfa05bb101380a8f0d41393de6ec7b06170529824d09"} Oct 31 00:26:34.602567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.602493 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podStartSLOduration=2.602474044 podStartE2EDuration="2.602474044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:34.600007821 +0000 UTC m=+1374.693298058" watchObservedRunningTime="2026-10-31 00:26:34.602474044 +0000 UTC m=+1374.695764232" Oct 31 00:26:34.954738 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.949311 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:34.954738 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.954295 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:26:34.954738 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:34.954352 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:26:35.585168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.585144 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:35.713858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.713776 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca\") pod \"1036eeff-51ee-40e4-ae45-fb725bb931dd\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " Oct 31 00:26:35.714029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.713893 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca\") pod \"1036eeff-51ee-40e4-ae45-fb725bb931dd\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " Oct 31 00:26:35.714029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.713962 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx\" (UniqueName: \"kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx\") pod \"1036eeff-51ee-40e4-ae45-fb725bb931dd\" (UID: \"1036eeff-51ee-40e4-ae45-fb725bb931dd\") " Oct 31 00:26:35.714328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.714259 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca" pod "1036eeff-51ee-40e4-ae45-fb725bb931dd" (UID: "1036eeff-51ee-40e4-ae45-fb725bb931dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:35.714531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.714511 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:35.716537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.716500 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca" pod "1036eeff-51ee-40e4-ae45-fb725bb931dd" (UID: "1036eeff-51ee-40e4-ae45-fb725bb931dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:35.717578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.717540 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx" pod "1036eeff-51ee-40e4-ae45-fb725bb931dd" (UID: "1036eeff-51ee-40e4-ae45-fb725bb931dd"). InnerVolumeSpecName "kube-api-access-g54lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:35.815763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.815731 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1036eeff-51ee-40e4-ae45-fb725bb931dd-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:35.815763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.815757 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54lx\" (UniqueName: \"kubernetes.io/projected/1036eeff-51ee-40e4-ae45-fb725bb931dd-kube-api-access-g54lx\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:35.953232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.949561 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:26:35.953232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:35.949627 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:26:36.060574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.060542 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7"] Oct 31 00:26:36.060574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.060579 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7011d03-bssc7"] Oct 31 00:26:36.376707 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.376628 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" Oct 31 00:26:36.376707 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.376669 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f" event={"ID":"1036eeff-51ee-40e4-ae45-fb725bb931dd","Type":"ContainerDied","Data":"9a39a977571909f6d75b80a246135e49361cb4976a773b1b339284d301a7bb16"} Oct 31 00:26:36.376943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.376714 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a39a977571909f6d75b80a246135e49361cb4976a773b1b339284d301a7bb16" Oct 31 00:26:36.948762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.948724 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:26:36.949473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:36.949429 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:26:37.857384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.857338 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z"] Oct 31 00:26:37.858283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.858257 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1036eeff-51ee-40e4-ae45-fb725bb931dd" containerName="revoke-keys" Oct 31 00:26:37.858464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.858437 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1036eeff-51ee-40e4-ae45-fb725bb931dd" containerName="revoke-keys" Oct 31 00:26:37.858726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.858709 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1036eeff-51ee-40e4-ae45-fb725bb931dd" containerName="revoke-keys" Oct 31 00:26:37.863922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.863901 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.957975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.957937 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:26:37.958480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.958020 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:26:37.964419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.964340 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z"] Oct 31 00:26:37.967875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.967848 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.967895 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.967990 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.968020 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.968047 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.968087 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.968357 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkls\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.968441 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:37.968895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:37.968475 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.062104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.062067 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af89437-140e-402d-ad56-bda892fda9c8" path="/var/lib/kubelet/pods/0af89437-140e-402d-ad56-bda892fda9c8/volumes" Oct 31 00:26:38.069447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069417 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069456 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069521 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069548 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069566 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069597 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069655 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkls\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069716 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069744 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.069943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069928 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.070044 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.069977 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.070123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.070097 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.070293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.070272 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.070405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.070378 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.072279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.072252 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.072450 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.072428 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.158539 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.158450 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.158842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.158819 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkls\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.181216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.181163 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:38.186044 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.186011 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:38.654216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.654143 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z"] Oct 31 00:26:38.689780 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:38.689747 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47ac0d7_d51b_4e0f_b17e_558e7ce33de5.slice/crio-8ccf9ddda0e065e2dbfae3ddab411352115634e85fbf8a35200faad885165e86 WatchSource:0}: Error finding container 8ccf9ddda0e065e2dbfae3ddab411352115634e85fbf8a35200faad885165e86: Status 404 returned error can't find the container with id 8ccf9ddda0e065e2dbfae3ddab411352115634e85fbf8a35200faad885165e86 Oct 31 00:26:38.692255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.692220 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:38.692341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.692293 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:38.692341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.692322 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:26:38.951263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.944559 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:26:38.951263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:38.944621 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:26:39.418777 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:39.418735 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" event={"ID":"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5","Type":"ContainerStarted","Data":"d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024"} Oct 31 00:26:39.419228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:39.418811 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" event={"ID":"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5","Type":"ContainerStarted","Data":"8ccf9ddda0e065e2dbfae3ddab411352115634e85fbf8a35200faad885165e86"} Oct 31 00:26:39.946462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:39.945242 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:26:39.946462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:39.945308 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:26:40.182505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:40.182476 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:40.184398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:40.184373 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:40.184542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:40.184417 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:40.956548 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:40.956513 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:40.957241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:40.956875 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:40.957920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:40.957864 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:26:41.154210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:41.153710 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podStartSLOduration=4.153690419 podStartE2EDuration="4.153690419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:39.54855809 +0000 UTC m=+1379.641848278" watchObservedRunningTime="2026-10-31 00:26:41.153690419 +0000 UTC m=+1381.246980607" Oct 31 00:26:41.185202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:41.184572 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:41.185202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:41.184645 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:42.203202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:42.182382 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:42.203202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:42.182444 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:43.035204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.023331 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.150314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.150270 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 00:26:43.187206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.186657 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:43.187206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.186719 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:43.205257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.205225 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z"] Oct 31 00:26:43.217587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.217556 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.226810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.224593 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.246498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.246462 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.247055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.247030 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.247253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.247235 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvlp\" (UniqueName: \"kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.281204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.281158 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:26:43.299255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.294624 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z"] Oct 31 00:26:43.332481 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.332450 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:26:43.332890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.332868 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:43.333051 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.333036 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:43.339201 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.333601 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.352887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.352839 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.353134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.353116 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvlp\" (UniqueName: \"kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.353336 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.353309 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.355996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.355964 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.450203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.449817 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.467209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.463129 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.504198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.493989 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvlp\" (UniqueName: \"kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp\") pod \"maas-api-revoke-sub-2e132981-97v5z\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:43.504198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.494090 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.550759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:43.550672 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:44.068612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.068583 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z"] Oct 31 00:26:44.070104 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:26:44.070072 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bba178_4f55_40f1_bc05_f06274a4155b.slice/crio-e00dbe901b144fe3748f94f8e1b7a76d61ae5e9956316712dd5447a940cfcced WatchSource:0}: Error finding container e00dbe901b144fe3748f94f8e1b7a76d61ae5e9956316712dd5447a940cfcced: Status 404 returned error can't find the container with id e00dbe901b144fe3748f94f8e1b7a76d61ae5e9956316712dd5447a940cfcced Oct 31 00:26:44.094120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.094091 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:44.182193 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.182148 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:44.182370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.182235 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:44.381569 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.381492 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:44.560378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.560333 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" event={"ID":"f7bba178-4f55-40f1-bc05-f06274a4155b","Type":"ContainerStarted","Data":"b859818ffd19eb7af2b5c41b2356b6c9e9686de5d1385fa95702ded0ec9062fc"} Oct 31 00:26:44.560593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:44.560467 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" event={"ID":"f7bba178-4f55-40f1-bc05-f06274a4155b","Type":"ContainerStarted","Data":"e00dbe901b144fe3748f94f8e1b7a76d61ae5e9956316712dd5447a940cfcced"} Oct 31 00:26:45.181811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:45.181780 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:45.181971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:45.181830 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:45.569404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:45.569326 5943 generic.go:356] "Generic (PLEG): container finished" podID="f7bba178-4f55-40f1-bc05-f06274a4155b" containerID="b859818ffd19eb7af2b5c41b2356b6c9e9686de5d1385fa95702ded0ec9062fc" exitCode=0 Oct 31 00:26:45.569404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:45.569401 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" event={"ID":"f7bba178-4f55-40f1-bc05-f06274a4155b","Type":"ContainerDied","Data":"b859818ffd19eb7af2b5c41b2356b6c9e9686de5d1385fa95702ded0ec9062fc"} Oct 31 00:26:45.753803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:45.753763 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:46.181693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.181660 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:26:46.181887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.181716 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:26:46.761070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.761034 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:46.827316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.827230 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp\" (UniqueName: \"kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp\") pod \"f7bba178-4f55-40f1-bc05-f06274a4155b\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " Oct 31 00:26:46.827502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.827439 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca\") pod \"f7bba178-4f55-40f1-bc05-f06274a4155b\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " Oct 31 00:26:46.827567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.827531 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca\") pod \"f7bba178-4f55-40f1-bc05-f06274a4155b\" (UID: \"f7bba178-4f55-40f1-bc05-f06274a4155b\") " Oct 31 00:26:46.827852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.827824 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca" pod "f7bba178-4f55-40f1-bc05-f06274a4155b" (UID: "f7bba178-4f55-40f1-bc05-f06274a4155b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:46.828018 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.828002 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.829493 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.829468 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp" pod "f7bba178-4f55-40f1-bc05-f06274a4155b" (UID: "f7bba178-4f55-40f1-bc05-f06274a4155b"). InnerVolumeSpecName "kube-api-access-whvlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.829631 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.829610 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca" pod "f7bba178-4f55-40f1-bc05-f06274a4155b" (UID: "f7bba178-4f55-40f1-bc05-f06274a4155b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:46.928778 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.928745 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bba178-4f55-40f1-bc05-f06274a4155b-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.928778 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:46.928776 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvlp\" (UniqueName: \"kubernetes.io/projected/f7bba178-4f55-40f1-bc05-f06274a4155b-kube-api-access-whvlp\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:47.587119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:47.587084 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" Oct 31 00:26:47.587119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:47.587106 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z" event={"ID":"f7bba178-4f55-40f1-bc05-f06274a4155b","Type":"ContainerDied","Data":"e00dbe901b144fe3748f94f8e1b7a76d61ae5e9956316712dd5447a940cfcced"} Oct 31 00:26:47.587437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:47.587168 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00dbe901b144fe3748f94f8e1b7a76d61ae5e9956316712dd5447a940cfcced" Oct 31 00:26:47.887893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:47.887811 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:47.888392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:47.888221 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:47.889041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:47.889017 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:26:48.916549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:48.916516 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.765405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:53.765368 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:55.440593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:55.440556 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:55.599927 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:55.599890 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:56.182233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:56.182173 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:26:56.846904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:56.846867 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:57.549636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:57.549535 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:57.585546 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:26:57.584376 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:26:58.137448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:58.137408 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:58.962471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:58.962397 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 31 00:26:59.776409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:26:59.765783 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:00.094426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.094304 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:00.824228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.820103 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf"] Oct 31 00:27:00.824228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.821011 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f7bba178-4f55-40f1-bc05-f06274a4155b" containerName="revoke-keys" Oct 31 00:27:00.824228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.821032 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bba178-4f55-40f1-bc05-f06274a4155b" containerName="revoke-keys" Oct 31 00:27:00.824228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.821424 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bba178-4f55-40f1-bc05-f06274a4155b" containerName="revoke-keys" Oct 31 00:27:00.825656 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.825635 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:00.893204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.887903 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0144bd-openshift-default-dockercfg-4hs4v\"" Oct 31 00:27:00.962210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.933993 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf"] Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:00.995677 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007435 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ccs\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007496 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007553 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007601 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007640 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007673 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007724 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.022425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.007785 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.111421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.109266 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.113399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.113370 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ccs\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.113961 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.113937 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114311 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114371 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114408 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114437 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114500 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114643 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114561 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.114977 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.114958 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.117425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.115930 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.117558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.112953 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.117650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.116442 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.117822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.117805 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.120157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.119695 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.123584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.122386 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.207205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.195951 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ccs\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.220464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.218654 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.306737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.306703 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:01.791695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.791656 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf"] Oct 31 00:27:01.826611 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:27:01.826580 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3218053c_1eb0_4bc8_9c6e_9654627540b2.slice/crio-059ead77f1e02caef55a9afd2f7d71d5a63e4c6e7aed8266295c3a3aefdfacf5 WatchSource:0}: Error finding container 059ead77f1e02caef55a9afd2f7d71d5a63e4c6e7aed8266295c3a3aefdfacf5: Status 404 returned error can't find the container with id 059ead77f1e02caef55a9afd2f7d71d5a63e4c6e7aed8266295c3a3aefdfacf5 Oct 31 00:27:01.828979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.828943 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:27:01.829077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.829014 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:27:01.829077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.829045 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:27:01.881614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:01.881582 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" event={"ID":"3218053c-1eb0-4bc8-9c6e-9654627540b2","Type":"ContainerStarted","Data":"059ead77f1e02caef55a9afd2f7d71d5a63e4c6e7aed8266295c3a3aefdfacf5"} Oct 31 00:27:02.809687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.809652 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw"] Oct 31 00:27:02.813745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.813724 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.890556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.890517 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" event={"ID":"3218053c-1eb0-4bc8-9c6e-9654627540b2","Type":"ContainerStarted","Data":"ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471"} Oct 31 00:27:02.894933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.894909 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.894940 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.894981 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.895127 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.895158 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.895288 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvbq\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.895333 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.895357 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.895476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.895413 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.963646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.963613 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw"] Oct 31 00:27:02.998949 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.998913 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.998949 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.998951 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.998991 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999043 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999071 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999141 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999257 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999293 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999386 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvbq\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999453 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999491 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:02.999945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:02.999919 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.000155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.000073 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.000263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.000223 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.002498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.002475 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.003716 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.003695 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.062502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.062426 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.064015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.063984 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvbq\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.126440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.126388 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podStartSLOduration=3.126374456 podStartE2EDuration="3.126374456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:03.123942666 +0000 UTC m=+1403.217232895" watchObservedRunningTime="2026-10-31 00:27:03.126374456 +0000 UTC m=+1403.219664643" Oct 31 00:27:03.151482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.151438 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:03.307668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.307629 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:03.308968 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.308939 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:27:03.309093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.309014 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 00:27:03.324932 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.324853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.553393 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.550933 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.592113 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.592088 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw"] Oct 31 00:27:03.594258 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:27:03.594235 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa478f7_2f67_48ff_b604_b0716fe8e55e.slice/crio-5a80aee0f7bcd870ed787372cb5654607aa32068cc3f2df00c4253b9156831ac WatchSource:0}: Error finding container 5a80aee0f7bcd870ed787372cb5654607aa32068cc3f2df00c4253b9156831ac: Status 404 returned error can't find the container with id 5a80aee0f7bcd870ed787372cb5654607aa32068cc3f2df00c4253b9156831ac Oct 31 00:27:03.596858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.596828 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:27:03.596937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.596896 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:27:03.596937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.596926 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:27:03.899263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.899140 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" event={"ID":"8aa478f7-2f67-48ff-b604-b0716fe8e55e","Type":"ContainerStarted","Data":"8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a"} Oct 31 00:27:03.899263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:03.899220 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" event={"ID":"8aa478f7-2f67-48ff-b604-b0716fe8e55e","Type":"ContainerStarted","Data":"5a80aee0f7bcd870ed787372cb5654607aa32068cc3f2df00c4253b9156831ac"} Oct 31 00:27:04.062050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.061987 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podStartSLOduration=2.061971582 podStartE2EDuration="2.061971582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:04.060562862 +0000 UTC m=+1404.153853067" watchObservedRunningTime="2026-10-31 00:27:04.061971582 +0000 UTC m=+1404.155261769" Oct 31 00:27:04.156477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.156373 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:04.161539 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.161504 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:04.161713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.161573 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:04.307376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.307344 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:27:04.307555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.307394 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 00:27:04.774204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.771392 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:04.945725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:04.945681 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.105535 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.105439 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.163203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.162582 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:05.163203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.162643 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:05.256200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.255817 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.307688 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.307656 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:27:05.307863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.307730 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 00:27:05.406879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:05.406787 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.162205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:06.160539 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:06.162205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:06.160600 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:06.307487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:06.307452 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:27:06.307658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:06.307504 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 00:27:06.684570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:06.684527 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.845206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:06.842101 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:07.007143 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:07.006782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:07.163395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:07.163357 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:07.163982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:07.163950 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:07.308216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:07.308114 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:27:07.308216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:07.308207 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 00:27:08.170429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:08.170391 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:08.171017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:08.170983 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:08.307581 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:08.307547 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:27:08.307753 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:08.307610 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 00:27:08.579931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:08.579887 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:08.779276 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:08.779170 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:09.158619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:09.158576 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:09.158843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:09.158645 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:09.813120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:09.813075 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:09.813609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:09.813588 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:09.814483 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:09.814463 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:27:10.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:10.156928 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:27:10.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:10.156983 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:27:10.784528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:10.784489 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:11.439432 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:11.439403 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:11.439976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:11.439742 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:11.440651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:11.440633 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:27:15.556916 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:15.556878 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:16.278362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:16.278323 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:20.495989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:20.495952 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:22.485395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.485329 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.765107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.764998 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.811920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.811862 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.845884 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.845845 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.929645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.929601 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.001304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.966458 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.059602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:22.966917 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 00:27:23.060407 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.060371 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.160759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.160707 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.183349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.183299 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.225802 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.224977 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.328268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.315605 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:23.457205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.370391 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.457205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.370488 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.524636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.522927 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.524636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.523462 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:27:23.797198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.797127 5943 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.847295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.847247 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.847919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.847895 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:27:23.993955 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.993885 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.994432 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:23.994394 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:27:24.002513 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.002467 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.003163 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.003138 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:27:24.072918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.072798 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.077214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.073611 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:27:24.237677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.237647 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 00:27:24.267392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.267303 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.308068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.308031 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:27:24.309325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.309305 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 00:27:24.310629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.310598 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 00:27:24.311784 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.311763 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:27:24.439312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.439170 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 00:27:24.439947 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.439927 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 00:27:24.449747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.449692 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 00:27:24.464253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.464202 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.465326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.465298 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:27:24.480271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.480241 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 00:27:24.612801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.612774 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 00:27:24.617292 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:24.615735 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 00:27:25.964346 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:25.964305 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:27.476361 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:27:27.476322 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:27:28.418140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:28.418107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:29.336301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:29.336264 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:30.896535 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:30.896498 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:34.585488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:34.585444 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:35.548283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:35.548249 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:35.742475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:35.742443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:35.892814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:35.892713 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:39.898012 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:39.897978 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:27:40.275814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:40.275778 5943 scope.go:122] "RemoveContainer" containerID="2519e521b443c008ea9b248cfdc3941a8f08979ebcd1eab45f1e074e03f99b59" Oct 31 00:27:40.582147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:40.582057 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:42.000726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:42.000320 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:27:42.000726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:42.000410 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.155208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:42.149307 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.314396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:42.314331 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.324655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:42.324605 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:45.902317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:45.902278 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:47.352820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:47.352781 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:50.740245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:50.740148 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:55.546080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:27:55.546044 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:57.477733 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:27:57.477684 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:28:00.364432 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:00.364388 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:05.145075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:05.145030 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.973923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:09.973876 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:12.490196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:12.490137 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:28:13.004567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.004530 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw"] Oct 31 00:28:13.008772 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.008755 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.092428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.092387 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:13.110674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.110631 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"distinct-2-w3-7abb95-kserve-self-signed-certs\"" Oct 31 00:28:13.113499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.113477 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw"] Oct 31 00:28:13.117370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.117346 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.117496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.117410 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.117610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.117591 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x49x8\" (UniqueName: \"kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.117689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.117672 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.117748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.117707 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.117985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.117963 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.219675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.219633 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.219932 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.219694 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.219932 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.219928 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.220118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.219970 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.220118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.220011 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.220118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.220049 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x49x8\" (UniqueName: \"kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.221693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.221666 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.221864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.221799 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.222273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.222247 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.224610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.224584 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.225878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.225844 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.295217 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.292111 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x49x8\" (UniqueName: \"kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8\") pod \"distinct-2-w3-7abb95-kserve-856866f5dc-774nw\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.326065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.325735 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:13.866235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.866111 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw"] Oct 31 00:28:13.871880 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:28:13.871850 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4635629_dc5b_4a7b_b985_06319473adcf.slice/crio-46fafdcf5a0759db13ff5dcabcd8f8b8b8133fb109517d1a6691b1cf6afad4cb WatchSource:0}: Error finding container 46fafdcf5a0759db13ff5dcabcd8f8b8b8133fb109517d1a6691b1cf6afad4cb: Status 404 returned error can't find the container with id 46fafdcf5a0759db13ff5dcabcd8f8b8b8133fb109517d1a6691b1cf6afad4cb Oct 31 00:28:13.925605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:13.925564 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" event={"ID":"f4635629-dc5b-4a7b-b985-06319473adcf","Type":"ContainerStarted","Data":"46fafdcf5a0759db13ff5dcabcd8f8b8b8133fb109517d1a6691b1cf6afad4cb"} Oct 31 00:28:14.938937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:14.938896 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" event={"ID":"f4635629-dc5b-4a7b-b985-06319473adcf","Type":"ContainerStarted","Data":"45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355"} Oct 31 00:28:15.058821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:15.058754 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" podStartSLOduration=3.058732439 podStartE2EDuration="3.058732439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:15.057547273 +0000 UTC m=+1475.150837488" watchObservedRunningTime="2026-10-31 00:28:15.058732439 +0000 UTC m=+1475.152022628" Oct 31 00:28:15.269534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:15.269492 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:15.326846 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:15.326819 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:16.467329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:16.467285 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.480477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:20.480440 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:22.326659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:22.326619 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:22.985592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:22.985553 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:23.327628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:23.327597 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:23.344505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:23.344483 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:23.701114 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:23.701008 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:24.014267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:24.014237 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 00:28:25.270680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:25.270646 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:26.992863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:26.992828 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:27.191975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:27.191937 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:27.478246 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:28:27.478211 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:28:30.240364 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:30.240324 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.417231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:30.413366 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.560032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:30.559978 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:35.359823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:35.359784 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:40.164837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:40.164803 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.955070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:44.955032 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:46.465413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:46.465377 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:49.768395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:49.768360 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:53.959793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:53.959748 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m"] Oct 31 00:28:53.964693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:53.964668 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.017204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.017168 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:28:54.017522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.017496 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:28:54.020353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.020328 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.020507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.020491 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.020579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.020520 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdw7w\" (UniqueName: \"kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.066871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.066839 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:28:54.067116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.066876 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:28:54.069688 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.069662 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m"] Oct 31 00:28:54.121710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.121662 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.121915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.121792 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.121915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.121818 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdw7w\" (UniqueName: \"kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.122717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.122690 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.125035 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.125013 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.183005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.182972 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdw7w\" (UniqueName: \"kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w\") pod \"maas-api-revoke-sub-ef2811c3-kbr4m\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.279630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.279588 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:54.645700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.645617 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:54.653940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:54.653915 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m"] Oct 31 00:28:54.655857 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:28:54.655833 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867a76a9_5d03_4fd2_9d63_8461aa340bc1.slice/crio-78ed3152c3ccbcc5b63345c530b6bdea2914f1f3f55966d2460fd33cb7cf00cd WatchSource:0}: Error finding container 78ed3152c3ccbcc5b63345c530b6bdea2914f1f3f55966d2460fd33cb7cf00cd: Status 404 returned error can't find the container with id 78ed3152c3ccbcc5b63345c530b6bdea2914f1f3f55966d2460fd33cb7cf00cd Oct 31 00:28:55.206223 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.206121 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" event={"ID":"867a76a9-5d03-4fd2-9d63-8461aa340bc1","Type":"ContainerStarted","Data":"3bc7ec587ed816e983420871e592a9f1893399b4c8343f990f04afa243685fd5"} Oct 31 00:28:55.206223 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.206198 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" event={"ID":"867a76a9-5d03-4fd2-9d63-8461aa340bc1","Type":"ContainerStarted","Data":"78ed3152c3ccbcc5b63345c530b6bdea2914f1f3f55966d2460fd33cb7cf00cd"} Oct 31 00:28:55.371017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.370964 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" podStartSLOduration=2.370950429 podStartE2EDuration="2.370950429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:55.368904674 +0000 UTC m=+1515.462194885" watchObservedRunningTime="2026-10-31 00:28:55.370950429 +0000 UTC m=+1515.464240617" Oct 31 00:28:55.842787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.842749 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g"] Oct 31 00:28:55.847487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.847466 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.907301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.907270 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g"] Oct 31 00:28:55.946482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.946451 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.946482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.946482 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.946662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.946589 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.946662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.946621 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.946727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.946689 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.946767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.946748 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrht\" (UniqueName: \"kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:55.952323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.952296 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f7c92a85\"/\"openshift-service-ca.crt\"" Oct 31 00:28:55.952323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.952321 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f7c92a85\"/\"e2e-lifecycle-model-f7c92a85-kserve-self-signed-certs\"" Oct 31 00:28:55.952504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.952413 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f7c92a85\"/\"kube-root-ca.crt\"" Oct 31 00:28:55.952541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.952514 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f7c92a85\"/\"default-dockercfg-89vwf\"" Oct 31 00:28:55.974880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:55.974850 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.047944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.047911 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.047987 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrht\" (UniqueName: \"kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048122 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048143 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048245 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048296 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048493 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048656 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.048679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.048670 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.050935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.050909 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.051239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.051156 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.113570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.113496 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrht\" (UniqueName: \"kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht\") pod \"e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.160136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.160106 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:56.217485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.217445 5943 generic.go:356] "Generic (PLEG): container finished" podID="867a76a9-5d03-4fd2-9d63-8461aa340bc1" containerID="3bc7ec587ed816e983420871e592a9f1893399b4c8343f990f04afa243685fd5" exitCode=0 Oct 31 00:28:56.217908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.217625 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" event={"ID":"867a76a9-5d03-4fd2-9d63-8461aa340bc1","Type":"ContainerDied","Data":"3bc7ec587ed816e983420871e592a9f1893399b4c8343f990f04afa243685fd5"} Oct 31 00:28:56.551680 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:28:56.551646 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda131f50e_c5e7_4f3f_a0d7_a89df8dd973e.slice/crio-7a179ecdc7b1d32a1cdc4f348e84dfbc065fb9ddc7f36804e688d2787a694fdb WatchSource:0}: Error finding container 7a179ecdc7b1d32a1cdc4f348e84dfbc065fb9ddc7f36804e688d2787a694fdb: Status 404 returned error can't find the container with id 7a179ecdc7b1d32a1cdc4f348e84dfbc065fb9ddc7f36804e688d2787a694fdb Oct 31 00:28:56.552341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:56.552317 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g"] Oct 31 00:28:57.225480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.225445 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" event={"ID":"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e","Type":"ContainerStarted","Data":"f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d"} Oct 31 00:28:57.225903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.225496 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" event={"ID":"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e","Type":"ContainerStarted","Data":"7a179ecdc7b1d32a1cdc4f348e84dfbc065fb9ddc7f36804e688d2787a694fdb"} Oct 31 00:28:57.353100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.353038 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" podStartSLOduration=2.353019449 podStartE2EDuration="2.353019449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:57.350699025 +0000 UTC m=+1517.443989214" watchObservedRunningTime="2026-10-31 00:28:57.353019449 +0000 UTC m=+1517.446309636" Oct 31 00:28:57.388399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.388369 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:57.467925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.467895 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca\") pod \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " Oct 31 00:28:57.468086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.468072 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca\") pod \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " Oct 31 00:28:57.468150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.468133 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w\" (UniqueName: \"kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w\") pod \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\" (UID: \"867a76a9-5d03-4fd2-9d63-8461aa340bc1\") " Oct 31 00:28:57.468317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.468291 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca" pod "867a76a9-5d03-4fd2-9d63-8461aa340bc1" (UID: "867a76a9-5d03-4fd2-9d63-8461aa340bc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:57.468775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.468687 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:28:57.470646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.470620 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w" pod "867a76a9-5d03-4fd2-9d63-8461aa340bc1" (UID: "867a76a9-5d03-4fd2-9d63-8461aa340bc1"). InnerVolumeSpecName "kube-api-access-vdw7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:57.470757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.470678 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca" pod "867a76a9-5d03-4fd2-9d63-8461aa340bc1" (UID: "867a76a9-5d03-4fd2-9d63-8461aa340bc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:57.479349 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:28:57.479281 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:28:57.569822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.569793 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867a76a9-5d03-4fd2-9d63-8461aa340bc1-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:28:57.569822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:57.569819 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdw7w\" (UniqueName: \"kubernetes.io/projected/867a76a9-5d03-4fd2-9d63-8461aa340bc1-kube-api-access-vdw7w\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:28:58.160640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:58.160609 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:28:58.232484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:58.232455 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" Oct 31 00:28:58.232958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:58.232452 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m" event={"ID":"867a76a9-5d03-4fd2-9d63-8461aa340bc1","Type":"ContainerDied","Data":"78ed3152c3ccbcc5b63345c530b6bdea2914f1f3f55966d2460fd33cb7cf00cd"} Oct 31 00:28:58.232958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:58.232581 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78ed3152c3ccbcc5b63345c530b6bdea2914f1f3f55966d2460fd33cb7cf00cd" Oct 31 00:28:58.889230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:58.889168 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:59.446941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:59.446898 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:59.596588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:59.596548 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:59.771206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:28:59.770903 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:04.585570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:04.585529 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:04.792763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:04.792731 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:04.946750 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:04.946661 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.105304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:05.105265 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:06.161135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:06.161103 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:29:06.174217 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:06.174168 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:29:06.293646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:06.293613 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:29:07.438336 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:07.438299 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:07.599438 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:07.599400 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:08.590668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:08.590638 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:08.755338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:08.755301 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.137483 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.137454 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6"] Oct 31 00:29:09.165766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.165724 5943 cpu_manager.go:414] "containerMap: removing container" podUID="867a76a9-5d03-4fd2-9d63-8461aa340bc1" containerName="revoke-keys" Oct 31 00:29:09.166057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.166038 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867a76a9-5d03-4fd2-9d63-8461aa340bc1" containerName="revoke-keys" Oct 31 00:29:09.166392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.166375 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="867a76a9-5d03-4fd2-9d63-8461aa340bc1" containerName="revoke-keys" Oct 31 00:29:09.176172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.175557 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.225837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.225061 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6"] Oct 31 00:29:09.225837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.225084 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.225837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.225595 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24vh\" (UniqueName: \"kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh\") pod \"payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.231483 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.231448 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-835921-dockercfg-s79h6\"" Oct 31 00:29:09.256150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.256117 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-835921\"" Oct 31 00:29:09.327111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.327072 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.327300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.327164 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24vh\" (UniqueName: \"kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh\") pod \"payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.327804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.327781 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.388480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.388405 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24vh\" (UniqueName: \"kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh\") pod \"payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.528407 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.528375 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:09.848741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.848707 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.906193 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:09.906152 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6"] Oct 31 00:29:09.907734 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:29:09.907704 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503141a3_2c97_4781_a631_5b74e280e75d.slice/crio-e022499b375a477c87c28cd1bd649554aa4935cce5c7bd1cb2fda9a37a8b6e5f WatchSource:0}: Error finding container e022499b375a477c87c28cd1bd649554aa4935cce5c7bd1cb2fda9a37a8b6e5f: Status 404 returned error can't find the container with id e022499b375a477c87c28cd1bd649554aa4935cce5c7bd1cb2fda9a37a8b6e5f Oct 31 00:29:10.316103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:10.316064 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" event={"ID":"503141a3-2c97-4781-a631-5b74e280e75d","Type":"ContainerStarted","Data":"ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b"} Oct 31 00:29:10.316314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:10.316122 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:10.316314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:10.316143 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" event={"ID":"503141a3-2c97-4781-a631-5b74e280e75d","Type":"ContainerStarted","Data":"e022499b375a477c87c28cd1bd649554aa4935cce5c7bd1cb2fda9a37a8b6e5f"} Oct 31 00:29:10.434491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:10.434436 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" podStartSLOduration=1.434419983 podStartE2EDuration="1.434419983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:10.431759823 +0000 UTC m=+1530.525050037" watchObservedRunningTime="2026-10-31 00:29:10.434419983 +0000 UTC m=+1530.527710215" Oct 31 00:29:10.938678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:10.938641 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:14.556309 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:14.556269 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.908364 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:16.908332 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk"] Oct 31 00:29:16.912568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:16.912552 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:16.964811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:16.964781 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7abb95-openshift-default-dockercfg-d8tvz\"" Oct 31 00:29:17.025425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.023616 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk"] Oct 31 00:29:17.028245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.028211 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.028398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.028278 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.028398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.028330 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.032332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.032295 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.032475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.032359 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.032590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.032565 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcw2\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.032656 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.032630 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.032769 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.032743 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.032881 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.032851 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.134531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.134494 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.134709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.134547 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.134758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.134724 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.134836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.134822 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.134892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.134845 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.134952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.134903 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcw2\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135010 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135539 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135591 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135615 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135620 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135254 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135629 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.135708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.135382 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.137634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.137609 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.137854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.137836 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.196118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.196046 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.196288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.196127 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcw2\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.225977 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.225954 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:17.663117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.663090 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk"] Oct 31 00:29:17.665107 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:29:17.665079 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb521294b_b616_441b_920f_4418e7855055.slice/crio-3d0183144bddad0ec0a69112b9ab71c91aa40b151c3c4e20a49e6678031cfe4d WatchSource:0}: Error finding container 3d0183144bddad0ec0a69112b9ab71c91aa40b151c3c4e20a49e6678031cfe4d: Status 404 returned error can't find the container with id 3d0183144bddad0ec0a69112b9ab71c91aa40b151c3c4e20a49e6678031cfe4d Oct 31 00:29:17.668032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.667998 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:29:17.668145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.668082 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:29:17.668145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:17.668129 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:29:18.370007 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:18.369973 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" event={"ID":"b521294b-b616-441b-920f-4418e7855055","Type":"ContainerStarted","Data":"969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890"} Oct 31 00:29:18.370405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:18.370035 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" event={"ID":"b521294b-b616-441b-920f-4418e7855055","Type":"ContainerStarted","Data":"3d0183144bddad0ec0a69112b9ab71c91aa40b151c3c4e20a49e6678031cfe4d"} Oct 31 00:29:18.490835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:18.490668 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podStartSLOduration=2.490651734 podStartE2EDuration="2.490651734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:18.490035131 +0000 UTC m=+1538.583325348" watchObservedRunningTime="2026-10-31 00:29:18.490651734 +0000 UTC m=+1538.583941922" Oct 31 00:29:19.226908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:19.226816 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:19.228326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:19.228286 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:19.228483 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:19.228361 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:19.364468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:19.364433 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:20.226870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:20.226836 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:20.227363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:20.226887 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:21.227306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:21.227276 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:21.227701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:21.227352 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:21.325528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:21.325499 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:29:22.227046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:22.227011 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:22.227248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:22.227084 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:23.226695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:23.226662 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:23.227081 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:23.226718 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:23.788146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:23.788112 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:29:24.226358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:24.226281 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:24.226358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:24.226329 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:24.325014 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:24.324979 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:25.230204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:25.229586 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:29:25.230204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:25.229671 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:29:26.486694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:26.480706 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:26.486694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:26.482795 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:26.530380 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:26.530351 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:29:27.476800 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:29:27.476759 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:29:29.246213 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:29.246141 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:34.244532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:34.244485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:39.050963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:39.050925 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:43.803837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:43.803802 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:43.998892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:43.998857 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:46.822786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:46.822750 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:48.512478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:48.512445 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:48.966640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:48.966543 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:49.263515 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:49.263476 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:49.601975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:49.601869 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:50.112207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:50.107656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:50.421661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:50.421557 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.140565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:54.140519 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.353561 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:54.353522 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:56.833211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.830988 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2"] Oct 31 00:29:56.838878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.838845 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:56.892399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.892374 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2"] Oct 31 00:29:56.937862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.937837 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:29:56.938067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.938049 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:56.938146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.938106 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:29:56.940428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.940413 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:29:56.983104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.983075 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:56.983270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.983115 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljz5\" (UniqueName: \"kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:56.983270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:56.983243 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.084391 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.084311 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.084552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.084480 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.084552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.084508 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljz5\" (UniqueName: \"kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.085115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.085087 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.086819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.086801 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.145478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.145443 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljz5\" (UniqueName: \"kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5\") pod \"maas-api-revoke-sub-dec38b57-9wrh2\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.152268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.152245 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:29:57.398577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.398485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:57.479939 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:29:57.479900 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:29:57.532405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.532265 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2"] Oct 31 00:29:57.534527 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:29:57.534490 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b6fd36_64e0_4c13_858f_bf1cf0bbb5ec.slice/crio-f4ed4de4bf3313fee1e9fd7819afa140432f75fb851260dd6540d5830e4eea25 WatchSource:0}: Error finding container f4ed4de4bf3313fee1e9fd7819afa140432f75fb851260dd6540d5830e4eea25: Status 404 returned error can't find the container with id f4ed4de4bf3313fee1e9fd7819afa140432f75fb851260dd6540d5830e4eea25 Oct 31 00:29:57.558467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.558436 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:57.786798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.786760 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" event={"ID":"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec","Type":"ContainerStarted","Data":"f4ed4de4bf3313fee1e9fd7819afa140432f75fb851260dd6540d5830e4eea25"} Oct 31 00:29:57.788635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.788018 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq"] Oct 31 00:29:57.792837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.792815 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:57.847583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.846851 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ccdde\"" Oct 31 00:29:57.847583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.847393 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ccdde-dockercfg-22jwj\"" Oct 31 00:29:57.852714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.852689 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq"] Oct 31 00:29:57.894655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.894626 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:57.894789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.894680 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fthw\" (UniqueName: \"kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw\") pod \"payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:57.996398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.996351 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:57.996580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.996427 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fthw\" (UniqueName: \"kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw\") pod \"payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:57.997518 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:57.997491 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:58.061694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.060470 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fthw\" (UniqueName: \"kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw\") pod \"payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:58.108297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.108264 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:58.489141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.489108 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq"] Oct 31 00:29:58.490965 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:29:58.490928 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f1ce65_d6e0_48bd_a53c_857d2f73239f.slice/crio-069105150b17b6c0a79d33b020b76a7964f49ba3fc0c9e6ba22a6ffd4254bb56 WatchSource:0}: Error finding container 069105150b17b6c0a79d33b020b76a7964f49ba3fc0c9e6ba22a6ffd4254bb56: Status 404 returned error can't find the container with id 069105150b17b6c0a79d33b020b76a7964f49ba3fc0c9e6ba22a6ffd4254bb56 Oct 31 00:29:58.794252 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.794215 5943 generic.go:356] "Generic (PLEG): container finished" podID="15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" containerID="605b7095697c40039fc8a492f44f633cee3f38527a65ae3e6d386905754d8b7f" exitCode=0 Oct 31 00:29:58.794428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.794302 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" event={"ID":"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec","Type":"ContainerDied","Data":"605b7095697c40039fc8a492f44f633cee3f38527a65ae3e6d386905754d8b7f"} Oct 31 00:29:58.795945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.795924 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" event={"ID":"42f1ce65-d6e0-48bd-a53c-857d2f73239f","Type":"ContainerStarted","Data":"abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96"} Oct 31 00:29:58.796070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.795971 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:29:58.796070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:58.795988 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" event={"ID":"42f1ce65-d6e0-48bd-a53c-857d2f73239f","Type":"ContainerStarted","Data":"069105150b17b6c0a79d33b020b76a7964f49ba3fc0c9e6ba22a6ffd4254bb56"} Oct 31 00:29:59.024297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:59.024150 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" podStartSLOduration=2.024132659 podStartE2EDuration="2.024132659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:59.02331532 +0000 UTC m=+1579.116605507" watchObservedRunningTime="2026-10-31 00:29:59.024132659 +0000 UTC m=+1579.117422846" Oct 31 00:29:59.173655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:59.173618 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:59.948206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:29:59.948159 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:30:00.022000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.021959 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca\") pod \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " Oct 31 00:30:00.022167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.022155 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca\") pod \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " Oct 31 00:30:00.022287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.022225 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5\" (UniqueName: \"kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5\") pod \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\" (UID: \"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec\") " Oct 31 00:30:00.022353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.022329 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca" pod "15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" (UID: "15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:00.022640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.022622 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:00.024400 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.024375 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca" pod "15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" (UID: "15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:00.024400 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.024385 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5" pod "15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" (UID: "15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec"). InnerVolumeSpecName "kube-api-access-sljz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:00.124436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.124360 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:00.124436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.124389 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljz5\" (UniqueName: \"kubernetes.io/projected/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec-kube-api-access-sljz5\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:00.166102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.166072 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk"] Oct 31 00:30:00.166837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.166807 5943 cpu_manager.go:414] "containerMap: removing container" podUID="15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" containerName="revoke-keys" Oct 31 00:30:00.166837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.166834 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" containerName="revoke-keys" Oct 31 00:30:00.167002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.166952 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" containerName="revoke-keys" Oct 31 00:30:00.170330 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.170314 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.179975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.179956 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4"] Oct 31 00:30:00.184477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.184461 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w"] Oct 31 00:30:00.184604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.184583 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.188357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.188340 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.225107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.225080 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.225294 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.225277 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5rt\" (UniqueName: \"kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.232425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.232391 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk"] Oct 31 00:30:00.237033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.237010 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk"] Oct 31 00:30:00.237135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.237122 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.294842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.294811 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w"] Oct 31 00:30:00.302322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.302295 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4"] Oct 31 00:30:00.322556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.322526 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk"] Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328084 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzth\" (UniqueName: \"kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth\") pod \"maas-api-key-cleanup-29890110-p7kq4\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328142 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5rt\" (UniqueName: \"kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328171 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gmr\" (UniqueName: \"kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr\") pod \"maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328243 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328320 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-p7kq4\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328363 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328435 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxh8c\" (UniqueName: \"kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.328455 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.329358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.329355 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.389888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.389821 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5rt\" (UniqueName: \"kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.429540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.429505 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxh8c\" (UniqueName: \"kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.429710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.429556 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.429710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.429643 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzth\" (UniqueName: \"kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth\") pod \"maas-api-key-cleanup-29890110-p7kq4\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.429710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.429677 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gmr\" (UniqueName: \"kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr\") pod \"maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.429854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.429727 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.429854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.429814 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-p7kq4\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.430488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.430466 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.430564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.430506 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.430609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.430582 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-p7kq4\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.483915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.483889 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:00.491239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.491212 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzth\" (UniqueName: \"kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth\") pod \"maas-api-key-cleanup-29890110-p7kq4\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.491856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.491833 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gmr\" (UniqueName: \"kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr\") pod \"maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.491971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.491957 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxh8c\" (UniqueName: \"kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.499579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.499559 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:00.505640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.505613 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:00.556411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.553971 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:00.809713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.809682 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" event={"ID":"15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec","Type":"ContainerDied","Data":"f4ed4de4bf3313fee1e9fd7819afa140432f75fb851260dd6540d5830e4eea25"} Oct 31 00:30:00.809713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.809703 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2" Oct 31 00:30:00.809927 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.809736 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ed4de4bf3313fee1e9fd7819afa140432f75fb851260dd6540d5830e4eea25" Oct 31 00:30:00.874695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.874672 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk"] Oct 31 00:30:00.875670 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:30:00.875645 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3da279_9433_4de9_af35_6abb16be7ad3.slice/crio-a72de2570e81138c40f281475eaecd140a709a686a218df99d89f7a5916ef5a5 WatchSource:0}: Error finding container a72de2570e81138c40f281475eaecd140a709a686a218df99d89f7a5916ef5a5: Status 404 returned error can't find the container with id a72de2570e81138c40f281475eaecd140a709a686a218df99d89f7a5916ef5a5 Oct 31 00:30:00.926053 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.926026 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w"] Oct 31 00:30:00.927973 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:30:00.927904 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b43c81a_d202_4f9d_99a2_7df3ba04a9d7.slice/crio-8747f75843daee1cbd142efe44090fce4d34d341192d2f9ded5217b141cf4a0c WatchSource:0}: Error finding container 8747f75843daee1cbd142efe44090fce4d34d341192d2f9ded5217b141cf4a0c: Status 404 returned error can't find the container with id 8747f75843daee1cbd142efe44090fce4d34d341192d2f9ded5217b141cf4a0c Oct 31 00:30:00.931313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.931288 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4"] Oct 31 00:30:00.932951 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:30:00.932918 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc73608d_0ab7_46ed_bc13_85387b1c1e93.slice/crio-3cc7723dde81607784df1d5995a036f347a7bf858bee7c27d8a51806510c4203 WatchSource:0}: Error finding container 3cc7723dde81607784df1d5995a036f347a7bf858bee7c27d8a51806510c4203: Status 404 returned error can't find the container with id 3cc7723dde81607784df1d5995a036f347a7bf858bee7c27d8a51806510c4203 Oct 31 00:30:00.975831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:00.975807 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk"] Oct 31 00:30:01.818504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.818461 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerStarted","Data":"36d1382fc93c0d9518bbab06bb344412299406a1d16678d32ae2bdac5d16d39c"} Oct 31 00:30:01.818977 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.818530 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerStarted","Data":"3cc7723dde81607784df1d5995a036f347a7bf858bee7c27d8a51806510c4203"} Oct 31 00:30:01.820238 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.820214 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" event={"ID":"25511e90-112e-474a-9227-253796b639c2","Type":"ContainerStarted","Data":"d6e64a236038fd912dfb42b2059e230b21c0da5532745f006ab0272b46735444"} Oct 31 00:30:01.820367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.820264 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" event={"ID":"25511e90-112e-474a-9227-253796b639c2","Type":"ContainerStarted","Data":"929e031549c3fbacab96ffab0575a4b9fdc3272a0278f1a06eb0fc4ce2f40ad7"} Oct 31 00:30:01.821860 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.821837 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerStarted","Data":"2ec58f9a657f21ee15aaaef26e5e2e34a7a496f52ee77006c3f7d47e620c7a6e"} Oct 31 00:30:01.821989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.821877 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerStarted","Data":"8747f75843daee1cbd142efe44090fce4d34d341192d2f9ded5217b141cf4a0c"} Oct 31 00:30:01.823353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.823332 5943 generic.go:356] "Generic (PLEG): container finished" podID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerID="b903c3498f039a3b87629f2f0a48df820ea517a9320b70122a3562c6c5c42640" exitCode=60 Oct 31 00:30:01.823419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.823366 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" event={"ID":"8c3da279-9433-4de9-af35-6abb16be7ad3","Type":"ContainerDied","Data":"b903c3498f039a3b87629f2f0a48df820ea517a9320b70122a3562c6c5c42640"} Oct 31 00:30:01.823419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.823403 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" event={"ID":"8c3da279-9433-4de9-af35-6abb16be7ad3","Type":"ContainerStarted","Data":"a72de2570e81138c40f281475eaecd140a709a686a218df99d89f7a5916ef5a5"} Oct 31 00:30:01.823612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.823597 5943 scope.go:122] "RemoveContainer" containerID="b903c3498f039a3b87629f2f0a48df820ea517a9320b70122a3562c6c5c42640" Oct 31 00:30:01.936026 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:01.935963 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" podStartSLOduration=1.935940803 podStartE2EDuration="1.935940803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:01.932904613 +0000 UTC m=+1582.026194795" watchObservedRunningTime="2026-10-31 00:30:01.935940803 +0000 UTC m=+1582.029230990" Oct 31 00:30:02.167032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.166989 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" podStartSLOduration=2.166973942 podStartE2EDuration="2.166973942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.165202333 +0000 UTC m=+1582.258492550" watchObservedRunningTime="2026-10-31 00:30:02.166973942 +0000 UTC m=+1582.260264161" Oct 31 00:30:02.284462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.284357 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" podStartSLOduration=2.284342578 podStartE2EDuration="2.284342578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.28211717 +0000 UTC m=+1582.375407381" watchObservedRunningTime="2026-10-31 00:30:02.284342578 +0000 UTC m=+1582.377632765" Oct 31 00:30:02.830966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.830928 5943 generic.go:356] "Generic (PLEG): container finished" podID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerID="36d1382fc93c0d9518bbab06bb344412299406a1d16678d32ae2bdac5d16d39c" exitCode=60 Oct 31 00:30:02.831388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.831020 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerDied","Data":"36d1382fc93c0d9518bbab06bb344412299406a1d16678d32ae2bdac5d16d39c"} Oct 31 00:30:02.831388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.831336 5943 scope.go:122] "RemoveContainer" containerID="36d1382fc93c0d9518bbab06bb344412299406a1d16678d32ae2bdac5d16d39c" Oct 31 00:30:02.832812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.832789 5943 generic.go:356] "Generic (PLEG): container finished" podID="25511e90-112e-474a-9227-253796b639c2" containerID="d6e64a236038fd912dfb42b2059e230b21c0da5532745f006ab0272b46735444" exitCode=60 Oct 31 00:30:02.832941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.832871 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" event={"ID":"25511e90-112e-474a-9227-253796b639c2","Type":"ContainerDied","Data":"d6e64a236038fd912dfb42b2059e230b21c0da5532745f006ab0272b46735444"} Oct 31 00:30:02.833145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.833129 5943 scope.go:122] "RemoveContainer" containerID="d6e64a236038fd912dfb42b2059e230b21c0da5532745f006ab0272b46735444" Oct 31 00:30:02.834693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.834532 5943 generic.go:356] "Generic (PLEG): container finished" podID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerID="2ec58f9a657f21ee15aaaef26e5e2e34a7a496f52ee77006c3f7d47e620c7a6e" exitCode=60 Oct 31 00:30:02.834693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.834612 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerDied","Data":"2ec58f9a657f21ee15aaaef26e5e2e34a7a496f52ee77006c3f7d47e620c7a6e"} Oct 31 00:30:02.834899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.834773 5943 scope.go:122] "RemoveContainer" containerID="2ec58f9a657f21ee15aaaef26e5e2e34a7a496f52ee77006c3f7d47e620c7a6e" Oct 31 00:30:02.837145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.837120 5943 generic.go:356] "Generic (PLEG): container finished" podID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerID="f3678bae36d2bdb3664dbdc3d424a8a76ad915505e4f8ef15ce92ae53536439d" exitCode=60 Oct 31 00:30:02.837264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.837171 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" event={"ID":"8c3da279-9433-4de9-af35-6abb16be7ad3","Type":"ContainerDied","Data":"f3678bae36d2bdb3664dbdc3d424a8a76ad915505e4f8ef15ce92ae53536439d"} Oct 31 00:30:02.837264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.837257 5943 scope.go:122] "RemoveContainer" containerID="b903c3498f039a3b87629f2f0a48df820ea517a9320b70122a3562c6c5c42640" Oct 31 00:30:02.837538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:02.837518 5943 scope.go:122] "RemoveContainer" containerID="f3678bae36d2bdb3664dbdc3d424a8a76ad915505e4f8ef15ce92ae53536439d" Oct 31 00:30:02.837776 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:02.837753 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk_odh-ai-gateway-infra(8c3da279-9433-4de9-af35-6abb16be7ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" Oct 31 00:30:03.846136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.846108 5943 scope.go:122] "RemoveContainer" containerID="f3678bae36d2bdb3664dbdc3d424a8a76ad915505e4f8ef15ce92ae53536439d" Oct 31 00:30:03.846552 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:03.846499 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk_odh-ai-gateway-infra(8c3da279-9433-4de9-af35-6abb16be7ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" Oct 31 00:30:03.847944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.847919 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerStarted","Data":"b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde"} Oct 31 00:30:03.848155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.848135 5943 scope.go:122] "RemoveContainer" containerID="b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde" Oct 31 00:30:03.848467 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:03.848446 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-p7kq4_odh-ai-gateway-infra(fc73608d-0ab7-46ed-bc13-85387b1c1e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" Oct 31 00:30:03.849885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.849865 5943 generic.go:356] "Generic (PLEG): container finished" podID="25511e90-112e-474a-9227-253796b639c2" containerID="5397058c9b20003e666a2ab770dab2ddc7d2f0c95459c43a57c9ae05dbc0bac3" exitCode=60 Oct 31 00:30:03.849975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.849940 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" event={"ID":"25511e90-112e-474a-9227-253796b639c2","Type":"ContainerDied","Data":"5397058c9b20003e666a2ab770dab2ddc7d2f0c95459c43a57c9ae05dbc0bac3"} Oct 31 00:30:03.850032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.849997 5943 scope.go:122] "RemoveContainer" containerID="d6e64a236038fd912dfb42b2059e230b21c0da5532745f006ab0272b46735444" Oct 31 00:30:03.850311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.850292 5943 scope.go:122] "RemoveContainer" containerID="5397058c9b20003e666a2ab770dab2ddc7d2f0c95459c43a57c9ae05dbc0bac3" Oct 31 00:30:03.850594 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:03.850573 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk_odh-ai-gateway-infra(25511e90-112e-474a-9227-253796b639c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" podUID="25511e90-112e-474a-9227-253796b639c2" Oct 31 00:30:03.851829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.851809 5943 generic.go:356] "Generic (PLEG): container finished" podID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerID="2bb44e903f9598c5ce0569f60719e053a8bebc28cd000da4d7509d05293cecbf" exitCode=60 Oct 31 00:30:03.851908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.851842 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerDied","Data":"2bb44e903f9598c5ce0569f60719e053a8bebc28cd000da4d7509d05293cecbf"} Oct 31 00:30:03.852071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.852058 5943 scope.go:122] "RemoveContainer" containerID="2bb44e903f9598c5ce0569f60719e053a8bebc28cd000da4d7509d05293cecbf" Oct 31 00:30:03.852319 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:03.852302 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w_odh-ai-gateway-infra(6b43c81a-d202-4f9d-99a2-7df3ba04a9d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" Oct 31 00:30:03.862110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:03.862095 5943 scope.go:122] "RemoveContainer" containerID="2ec58f9a657f21ee15aaaef26e5e2e34a7a496f52ee77006c3f7d47e620c7a6e" Oct 31 00:30:04.079443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.079396 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:04.860235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.860198 5943 generic.go:356] "Generic (PLEG): container finished" podID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerID="b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde" exitCode=60 Oct 31 00:30:04.860725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.860291 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerDied","Data":"b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde"} Oct 31 00:30:04.860725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.860352 5943 scope.go:122] "RemoveContainer" containerID="36d1382fc93c0d9518bbab06bb344412299406a1d16678d32ae2bdac5d16d39c" Oct 31 00:30:04.860725 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.860567 5943 scope.go:122] "RemoveContainer" containerID="b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde" Oct 31 00:30:04.860949 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:04.860924 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-p7kq4_odh-ai-gateway-infra(fc73608d-0ab7-46ed-bc13-85387b1c1e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" Oct 31 00:30:04.862668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.862645 5943 scope.go:122] "RemoveContainer" containerID="5397058c9b20003e666a2ab770dab2ddc7d2f0c95459c43a57c9ae05dbc0bac3" Oct 31 00:30:04.862976 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:04.862955 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk_odh-ai-gateway-infra(25511e90-112e-474a-9227-253796b639c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" podUID="25511e90-112e-474a-9227-253796b639c2" Oct 31 00:30:04.864343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:04.864328 5943 scope.go:122] "RemoveContainer" containerID="2bb44e903f9598c5ce0569f60719e053a8bebc28cd000da4d7509d05293cecbf" Oct 31 00:30:04.864598 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:04.864580 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w_odh-ai-gateway-infra(6b43c81a-d202-4f9d-99a2-7df3ba04a9d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" Oct 31 00:30:05.873868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:05.873841 5943 scope.go:122] "RemoveContainer" containerID="b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde" Oct 31 00:30:05.874285 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:05.874147 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-p7kq4_odh-ai-gateway-infra(fc73608d-0ab7-46ed-bc13-85387b1c1e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" Oct 31 00:30:08.868717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:08.868681 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:09.266288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.266258 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt"] Oct 31 00:30:09.271658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.271638 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.324461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.324417 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:30:09.324648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.324629 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g5g6j\"" Oct 31 00:30:09.324723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.324628 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:30:09.324723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.324652 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:30:09.377483 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.377456 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt"] Oct 31 00:30:09.452742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.452700 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.452925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.452770 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.452925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.452848 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.453100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.453079 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.453160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.453120 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.453212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.453172 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64d7x\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.453245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.453214 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554438 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554494 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554522 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554679 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554706 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554736 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64d7x\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.554969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.554869 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.555429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.555368 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.556902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.556870 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.557194 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.557159 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.557271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.557213 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.557385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.557368 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.629538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.629508 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.629959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.629935 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64d7x\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lvhqt\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:09.804621 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.804542 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:30:09.884793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:09.884766 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:10.323367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:10.323336 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt"] Oct 31 00:30:10.355775 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:30:10.355746 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode408554a_a619_44f0_ace6_afe9c9188a31.slice/crio-0a0911478f565f0fcb82df91309879960b5769672a589f2ddd5ea455a78ce767 WatchSource:0}: Error finding container 0a0911478f565f0fcb82df91309879960b5769672a589f2ddd5ea455a78ce767: Status 404 returned error can't find the container with id 0a0911478f565f0fcb82df91309879960b5769672a589f2ddd5ea455a78ce767 Oct 31 00:30:10.359872 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:10.359852 5943 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:30:10.907898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:10.907863 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" event={"ID":"e408554a-a619-44f0-ace6-afe9c9188a31","Type":"ContainerStarted","Data":"0a0911478f565f0fcb82df91309879960b5769672a589f2ddd5ea455a78ce767"} Oct 31 00:30:11.797049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:11.797019 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:30:13.840744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:13.840684 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:14.048250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:14.048219 5943 scope.go:122] "RemoveContainer" containerID="f3678bae36d2bdb3664dbdc3d424a8a76ad915505e4f8ef15ce92ae53536439d" Oct 31 00:30:14.984858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:14.984819 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" event={"ID":"8c3da279-9433-4de9-af35-6abb16be7ad3","Type":"ContainerStarted","Data":"1d3b30384db4c3c6af7dba124bc5951b7480e58afff733b76d80ac8926256fc3"} Oct 31 00:30:15.005506 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.005470 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:30:15.005651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.005550 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:30:15.035077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.035054 5943 scope.go:122] "RemoveContainer" containerID="b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde" Oct 31 00:30:15.104715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.104646 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" podStartSLOduration=15.104632123 podStartE2EDuration="15.104632123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:15.10096043 +0000 UTC m=+1595.194250626" watchObservedRunningTime="2026-10-31 00:30:15.104632123 +0000 UTC m=+1595.197922345" Oct 31 00:30:15.993520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.993476 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerStarted","Data":"8f84ef6cd00fd166b11e0dec4dbe4e2ba044a0638c1b03312037bc327db6e8a5"} Oct 31 00:30:15.995347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.995308 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" event={"ID":"e408554a-a619-44f0-ace6-afe9c9188a31","Type":"ContainerStarted","Data":"b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1"} Oct 31 00:30:15.995496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.995374 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:15.997443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.997417 5943 generic.go:356] "Generic (PLEG): container finished" podID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerID="1d3b30384db4c3c6af7dba124bc5951b7480e58afff733b76d80ac8926256fc3" exitCode=60 Oct 31 00:30:15.997550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.997475 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" event={"ID":"8c3da279-9433-4de9-af35-6abb16be7ad3","Type":"ContainerDied","Data":"1d3b30384db4c3c6af7dba124bc5951b7480e58afff733b76d80ac8926256fc3"} Oct 31 00:30:15.997550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.997535 5943 scope.go:122] "RemoveContainer" containerID="f3678bae36d2bdb3664dbdc3d424a8a76ad915505e4f8ef15ce92ae53536439d" Oct 31 00:30:15.997834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:15.997812 5943 scope.go:122] "RemoveContainer" containerID="1d3b30384db4c3c6af7dba124bc5951b7480e58afff733b76d80ac8926256fc3" Oct 31 00:30:15.998115 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:15.998085 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk_odh-ai-gateway-infra(8c3da279-9433-4de9-af35-6abb16be7ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" Oct 31 00:30:16.116939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:16.116905 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk"] Oct 31 00:30:16.235013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:16.234944 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" podStartSLOduration=2.59101201 podStartE2EDuration="7.234924102s" totalImagesPullingTime="4.643912092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:16.229230252 +0000 UTC m=+1596.322520464" watchObservedRunningTime="2026-10-31 00:30:16.234924102 +0000 UTC m=+1596.328214294" Oct 31 00:30:17.017712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.017684 5943 generic.go:356] "Generic (PLEG): container finished" podID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerID="8f84ef6cd00fd166b11e0dec4dbe4e2ba044a0638c1b03312037bc327db6e8a5" exitCode=60 Oct 31 00:30:17.018122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.017783 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerDied","Data":"8f84ef6cd00fd166b11e0dec4dbe4e2ba044a0638c1b03312037bc327db6e8a5"} Oct 31 00:30:17.018122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.017890 5943 scope.go:122] "RemoveContainer" containerID="b185da11fe8557ad675c003e996bbd4e3bf9c997cae304aeb0cef3b3bc6dacde" Oct 31 00:30:17.018520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.018501 5943 scope.go:122] "RemoveContainer" containerID="8f84ef6cd00fd166b11e0dec4dbe4e2ba044a0638c1b03312037bc327db6e8a5" Oct 31 00:30:17.018893 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:17.018866 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-p7kq4_odh-ai-gateway-infra(fc73608d-0ab7-46ed-bc13-85387b1c1e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" Oct 31 00:30:17.020372 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.020353 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:30:17.034567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.034551 5943 scope.go:122] "RemoveContainer" containerID="2bb44e903f9598c5ce0569f60719e053a8bebc28cd000da4d7509d05293cecbf" Oct 31 00:30:17.123271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.123239 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4"] Oct 31 00:30:17.264909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.264874 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:17.374101 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.373982 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt\" (UniqueName: \"kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt\") pod \"8c3da279-9433-4de9-af35-6abb16be7ad3\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " Oct 31 00:30:17.374825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.374794 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca\") pod \"8c3da279-9433-4de9-af35-6abb16be7ad3\" (UID: \"8c3da279-9433-4de9-af35-6abb16be7ad3\") " Oct 31 00:30:17.375993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.375956 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca" pod "8c3da279-9433-4de9-af35-6abb16be7ad3" (UID: "8c3da279-9433-4de9-af35-6abb16be7ad3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:17.377114 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.377089 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt" pod "8c3da279-9433-4de9-af35-6abb16be7ad3" (UID: "8c3da279-9433-4de9-af35-6abb16be7ad3"). InnerVolumeSpecName "kube-api-access-fv5rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:17.476988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.476960 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5rt\" (UniqueName: \"kubernetes.io/projected/8c3da279-9433-4de9-af35-6abb16be7ad3-kube-api-access-fv5rt\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.476988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:17.476989 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da279-9433-4de9-af35-6abb16be7ad3-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.025361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.025334 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" Oct 31 00:30:18.025361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.025353 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk" event={"ID":"8c3da279-9433-4de9-af35-6abb16be7ad3","Type":"ContainerDied","Data":"a72de2570e81138c40f281475eaecd140a709a686a218df99d89f7a5916ef5a5"} Oct 31 00:30:18.025893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.025409 5943 scope.go:122] "RemoveContainer" containerID="1d3b30384db4c3c6af7dba124bc5951b7480e58afff733b76d80ac8926256fc3" Oct 31 00:30:18.029859 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.029821 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerStarted","Data":"8d4d41016f79cad0e8d9e10bf1700b11804f8e6960071d19fdbb9104e8e46ebf"} Oct 31 00:30:18.034642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.034544 5943 scope.go:122] "RemoveContainer" containerID="5397058c9b20003e666a2ab770dab2ddc7d2f0c95459c43a57c9ae05dbc0bac3" Oct 31 00:30:18.190958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.190928 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk"] Oct 31 00:30:18.196832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.196808 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890110-55nqk"] Oct 31 00:30:18.319650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.319626 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:18.390675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.390642 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca\") pod \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " Oct 31 00:30:18.390809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.390758 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth\" (UniqueName: \"kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth\") pod \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\" (UID: \"fc73608d-0ab7-46ed-bc13-85387b1c1e93\") " Oct 31 00:30:18.390970 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.390943 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca" pod "fc73608d-0ab7-46ed-bc13-85387b1c1e93" (UID: "fc73608d-0ab7-46ed-bc13-85387b1c1e93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:18.391258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.391244 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73608d-0ab7-46ed-bc13-85387b1c1e93-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.428798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.428759 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth" pod "fc73608d-0ab7-46ed-bc13-85387b1c1e93" (UID: "fc73608d-0ab7-46ed-bc13-85387b1c1e93"). InnerVolumeSpecName "kube-api-access-cbzth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:18.492910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.492880 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbzth\" (UniqueName: \"kubernetes.io/projected/fc73608d-0ab7-46ed-bc13-85387b1c1e93-kube-api-access-cbzth\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.841806 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:18.841762 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:19.037426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.037396 5943 generic.go:356] "Generic (PLEG): container finished" podID="25511e90-112e-474a-9227-253796b639c2" containerID="17d65d3968bc23e604117c110d9000b7c42dd5ec28f9debdd8d51f3e92c48f4d" exitCode=60 Oct 31 00:30:19.037864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.037469 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" event={"ID":"25511e90-112e-474a-9227-253796b639c2","Type":"ContainerDied","Data":"17d65d3968bc23e604117c110d9000b7c42dd5ec28f9debdd8d51f3e92c48f4d"} Oct 31 00:30:19.037864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.037523 5943 scope.go:122] "RemoveContainer" containerID="5397058c9b20003e666a2ab770dab2ddc7d2f0c95459c43a57c9ae05dbc0bac3" Oct 31 00:30:19.037976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.037897 5943 scope.go:122] "RemoveContainer" containerID="17d65d3968bc23e604117c110d9000b7c42dd5ec28f9debdd8d51f3e92c48f4d" Oct 31 00:30:19.038304 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:19.038278 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk_odh-ai-gateway-infra(25511e90-112e-474a-9227-253796b639c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" podUID="25511e90-112e-474a-9227-253796b639c2" Oct 31 00:30:19.039617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.039596 5943 generic.go:356] "Generic (PLEG): container finished" podID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerID="8d4d41016f79cad0e8d9e10bf1700b11804f8e6960071d19fdbb9104e8e46ebf" exitCode=60 Oct 31 00:30:19.039717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.039669 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerDied","Data":"8d4d41016f79cad0e8d9e10bf1700b11804f8e6960071d19fdbb9104e8e46ebf"} Oct 31 00:30:19.039994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.039976 5943 scope.go:122] "RemoveContainer" containerID="8d4d41016f79cad0e8d9e10bf1700b11804f8e6960071d19fdbb9104e8e46ebf" Oct 31 00:30:19.040349 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:19.040329 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w_odh-ai-gateway-infra(6b43c81a-d202-4f9d-99a2-7df3ba04a9d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" Oct 31 00:30:19.041962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.041942 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" Oct 31 00:30:19.042056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.041946 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4" event={"ID":"fc73608d-0ab7-46ed-bc13-85387b1c1e93","Type":"ContainerDied","Data":"3cc7723dde81607784df1d5995a036f347a7bf858bee7c27d8a51806510c4203"} Oct 31 00:30:19.050296 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.050279 5943 scope.go:122] "RemoveContainer" containerID="2bb44e903f9598c5ce0569f60719e053a8bebc28cd000da4d7509d05293cecbf" Oct 31 00:30:19.063158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.063129 5943 scope.go:122] "RemoveContainer" containerID="8f84ef6cd00fd166b11e0dec4dbe4e2ba044a0638c1b03312037bc327db6e8a5" Oct 31 00:30:19.264727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.264693 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4"] Oct 31 00:30:19.269639 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.269609 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-p7kq4"] Oct 31 00:30:19.434803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:19.434771 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w"] Oct 31 00:30:20.042789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.042756 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" path="/var/lib/kubelet/pods/8c3da279-9433-4de9-af35-6abb16be7ad3/volumes" Oct 31 00:30:20.043415 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.043394 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" path="/var/lib/kubelet/pods/fc73608d-0ab7-46ed-bc13-85387b1c1e93/volumes" Oct 31 00:30:20.189798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.189768 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk"] Oct 31 00:30:20.206607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.206569 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:20.316485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.316404 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c\" (UniqueName: \"kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c\") pod \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " Oct 31 00:30:20.316649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.316526 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca\") pod \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\" (UID: \"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7\") " Oct 31 00:30:20.316968 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.316935 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca" pod "6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" (UID: "6b43c81a-d202-4f9d-99a2-7df3ba04a9d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:20.317567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.317544 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.319306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.319266 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c" pod "6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" (UID: "6b43c81a-d202-4f9d-99a2-7df3ba04a9d7"). InnerVolumeSpecName "kube-api-access-nxh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:20.358580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.358554 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:20.418367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.418342 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca\") pod \"25511e90-112e-474a-9227-253796b639c2\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " Oct 31 00:30:20.418534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.418476 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr\" (UniqueName: \"kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr\") pod \"25511e90-112e-474a-9227-253796b639c2\" (UID: \"25511e90-112e-474a-9227-253796b639c2\") " Oct 31 00:30:20.418685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.418662 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca" pod "25511e90-112e-474a-9227-253796b639c2" (UID: "25511e90-112e-474a-9227-253796b639c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:20.418905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.418888 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25511e90-112e-474a-9227-253796b639c2-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.418950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.418907 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxh8c\" (UniqueName: \"kubernetes.io/projected/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7-kube-api-access-nxh8c\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.420900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.420876 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr" pod "25511e90-112e-474a-9227-253796b639c2" (UID: "25511e90-112e-474a-9227-253796b639c2"). InnerVolumeSpecName "kube-api-access-s2gmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:20.520003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:20.519973 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gmr\" (UniqueName: \"kubernetes.io/projected/25511e90-112e-474a-9227-253796b639c2-kube-api-access-s2gmr\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:21.060745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.060708 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" Oct 31 00:30:21.061244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.060704 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk" event={"ID":"25511e90-112e-474a-9227-253796b639c2","Type":"ContainerDied","Data":"929e031549c3fbacab96ffab0575a4b9fdc3272a0278f1a06eb0fc4ce2f40ad7"} Oct 31 00:30:21.061244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.060808 5943 scope.go:122] "RemoveContainer" containerID="17d65d3968bc23e604117c110d9000b7c42dd5ec28f9debdd8d51f3e92c48f4d" Oct 31 00:30:21.063720 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.063685 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" Oct 31 00:30:21.063848 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.063727 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w" event={"ID":"6b43c81a-d202-4f9d-99a2-7df3ba04a9d7","Type":"ContainerDied","Data":"8747f75843daee1cbd142efe44090fce4d34d341192d2f9ded5217b141cf4a0c"} Oct 31 00:30:21.074283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.074209 5943 scope.go:122] "RemoveContainer" containerID="8d4d41016f79cad0e8d9e10bf1700b11804f8e6960071d19fdbb9104e8e46ebf" Oct 31 00:30:21.228540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.228505 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w"] Oct 31 00:30:21.234598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.234560 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890110-jn95w"] Oct 31 00:30:21.386001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.385915 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk"] Oct 31 00:30:21.390609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:21.390576 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c92a85-29890110-52mwk"] Oct 31 00:30:22.042918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:22.042881 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25511e90-112e-474a-9227-253796b639c2" path="/var/lib/kubelet/pods/25511e90-112e-474a-9227-253796b639c2/volumes" Oct 31 00:30:22.043571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:22.043543 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" path="/var/lib/kubelet/pods/6b43c81a-d202-4f9d-99a2-7df3ba04a9d7/volumes" Oct 31 00:30:23.756129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:23.756092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.477544 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:27.477503 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:30:28.642622 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:28.642587 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:32.506344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506313 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw"] Oct 31 00:30:32.506871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506858 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.506914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506872 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.506914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506899 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:30:32.506914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506904 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:30:32.506914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506914 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506920 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506935 5943 cpu_manager.go:414] "containerMap: removing container" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506940 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506947 5943 cpu_manager.go:414] "containerMap: removing container" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506952 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506963 5943 cpu_manager.go:414] "containerMap: removing container" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506967 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506975 5943 cpu_manager.go:414] "containerMap: removing container" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506980 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506993 5943 cpu_manager.go:414] "containerMap: removing container" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.506998 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507006 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507011 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507016 5943 cpu_manager.go:414] "containerMap: removing container" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507020 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507097 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507103 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507110 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507115 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507120 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507127 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507135 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507141 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507148 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="25511e90-112e-474a-9227-253796b639c2" containerName="cleanup" Oct 31 00:30:32.507516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.507155 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:30:32.510690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.510675 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577581 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577628 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577733 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577780 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sbl\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577856 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577894 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.577988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.577966 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.578093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.578079 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.578135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.578122 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.617923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.617884 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw"] Oct 31 00:30:32.679323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679279 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679347 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679383 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679412 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679456 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679494 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sbl\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679536 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679592 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679621 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679740 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.679945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.679806 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.680103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.680079 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.680266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.680248 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.680564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.680536 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.682835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.682811 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.683130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.683108 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.740648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.740619 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.740759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.740723 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sbl\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:32.825031 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:32.824958 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:33.265844 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:30:33.265066 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9999ba88_9551_4b1d_b54b_87b509350556.slice/crio-33e20c24497bc60dc98a10e82ae92d71f81c54c55a92cc9ccb4f0a3effefe241 WatchSource:0}: Error finding container 33e20c24497bc60dc98a10e82ae92d71f81c54c55a92cc9ccb4f0a3effefe241: Status 404 returned error can't find the container with id 33e20c24497bc60dc98a10e82ae92d71f81c54c55a92cc9ccb4f0a3effefe241 Oct 31 00:30:33.268152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:33.268117 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw"] Oct 31 00:30:33.270935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:33.270900 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:30:33.271019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:33.271000 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:30:33.271071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:33.271059 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:30:34.055207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.054712 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:34.170964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.170055 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" event={"ID":"9999ba88-9551-4b1d-b54b-87b509350556","Type":"ContainerStarted","Data":"cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc"} Oct 31 00:30:34.170964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.170111 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" event={"ID":"9999ba88-9551-4b1d-b54b-87b509350556","Type":"ContainerStarted","Data":"33e20c24497bc60dc98a10e82ae92d71f81c54c55a92cc9ccb4f0a3effefe241"} Oct 31 00:30:34.295154 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.295093 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podStartSLOduration=2.295077715 podStartE2EDuration="2.295077715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:34.291092861 +0000 UTC m=+1614.384383071" watchObservedRunningTime="2026-10-31 00:30:34.295077715 +0000 UTC m=+1614.388367893" Oct 31 00:30:34.826090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.826058 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:34.827794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.827765 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:30:34.827938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:34.827828 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:30:35.271329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:35.271294 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:35.825685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:35.825652 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:30:35.825925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:35.825706 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:30:36.170259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:36.165217 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.457254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:36.456852 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.622611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:36.622569 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.825917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:36.825879 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:30:36.826086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:36.825960 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:30:37.191275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:37.191200 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:37.834207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:37.830678 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:30:37.834207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:37.830753 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:30:37.863203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:37.858007 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:38.825428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:38.825390 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:30:38.825593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:38.825439 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:30:38.850205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:38.849961 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:39.268211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:39.268150 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:39.433582 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:39.433542 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:39.703603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:39.703509 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:39.826315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:39.826283 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:30:39.826486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:39.826336 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:30:40.863236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:40.863204 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:40.863791 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:40.863506 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:40.865089 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:40.865066 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:30:41.062553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:41.062515 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.265980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:41.265945 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.522172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:41.522083 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.676521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:41.676481 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.833396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:41.833296 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.765640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:43.765602 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.975463 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:43.975426 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:44.708858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:44.708819 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:45.287367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:45.287325 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:45.489593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:45.489558 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.203912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:47.203878 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.405563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:47.405527 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.943842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:47.943810 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:48.612036 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:48.611997 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:48.763331 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:48.763281 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:48.820768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:48.820686 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:30:49.053448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:49.053404 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr"] Oct 31 00:30:49.056385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:49.056360 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71460944-qdtgr"] Oct 31 00:30:49.853574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:49.853532 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:50.042466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:50.042434 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57616b1b-74a1-45da-87e0-55e6007ad9b4" path="/var/lib/kubelet/pods/57616b1b-74a1-45da-87e0-55e6007ad9b4/volumes" Oct 31 00:30:50.617764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:50.617730 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:50.797869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:50.797794 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.221091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:52.221056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.387918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:52.387880 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.555858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:52.555815 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:54.969447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:54.969413 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:57.479046 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:30:57.478996 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:30:57.912039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:57.911995 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:58.106522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:30:58.106461 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:00.249744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:00.249705 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:05.144127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:05.144079 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:09.869527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:09.869485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:10.048086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:10.048053 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r"] Oct 31 00:31:10.097259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:10.097229 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7cf7775-krb6r"] Oct 31 00:31:12.051311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:12.051266 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5459a02-388c-4c60-8376-f937b05a68de" path="/var/lib/kubelet/pods/b5459a02-388c-4c60-8376-f937b05a68de/volumes" Oct 31 00:31:14.721886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:14.721847 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:19.436486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:19.436452 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:24.062163 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:24.062129 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:24.206735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:24.206702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:24.411803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:24.411706 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.482406 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:31:27.482368 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:31:28.932137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:28.932100 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:30.559009 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:30.558965 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:31:33.789191 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:33.789143 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:33.947765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:33.947730 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:34.095107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:34.095003 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:36.024683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:36.024633 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:36.051573 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:36.051544 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f"] Oct 31 00:31:36.100937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:36.100901 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d547547-smk9f"] Oct 31 00:31:36.282095 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:36.281986 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:38.048351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:38.048312 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1036eeff-51ee-40e4-ae45-fb725bb931dd" path="/var/lib/kubelet/pods/1036eeff-51ee-40e4-ae45-fb725bb931dd/volumes" Oct 31 00:31:39.088001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:39.087967 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:44.334834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:44.334799 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:47.094537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:47.094496 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z"] Oct 31 00:31:47.097344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:47.097313 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e132981-97v5z"] Oct 31 00:31:47.547832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:47.547792 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:48.042008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:48.041968 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bba178-4f55-40f1-bc05-f06274a4155b" path="/var/lib/kubelet/pods/f7bba178-4f55-40f1-bc05-f06274a4155b/volumes" Oct 31 00:31:49.200887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:49.200857 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:49.365495 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:49.365459 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:49.561044 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:49.561008 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:52.815675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:52.815627 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:53.010791 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:53.010757 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:54.248814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:54.248781 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:57.476934 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:31:57.476884 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:31:59.032439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:31:59.032399 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:03.255281 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:03.255239 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:04.896879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:04.896841 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:05.041864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:05.041832 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:05.277244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:05.277164 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:09.506030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:09.505990 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:10.191351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:10.191318 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:14.948601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:14.948564 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:17.574637 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:17.574599 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:17.748248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:17.748213 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:19.907200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:19.907091 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:20.075009 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:20.074790 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:20.225654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:20.225553 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:22.239213 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:22.239164 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:32:25.145121 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:25.145088 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:27.477400 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:32:27.477361 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:32:28.213157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:28.213123 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:28.965918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:28.965873 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:29.573935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.573900 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd"] Oct 31 00:32:29.574500 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.574485 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:32:29.574550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.574501 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:32:29.574550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.574544 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:32:29.574628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.574549 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b43c81a-d202-4f9d-99a2-7df3ba04a9d7" containerName="cleanup" Oct 31 00:32:29.574674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.574665 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73608d-0ab7-46ed-bc13-85387b1c1e93" containerName="cleanup" Oct 31 00:32:29.574712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.574676 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3da279-9433-4de9-af35-6abb16be7ad3" containerName="cleanup" Oct 31 00:32:29.578206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.578170 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.634854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.634816 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2cda64-openshift-default-dockercfg-mdhw9\"" Oct 31 00:32:29.669208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.669143 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:29.688501 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.688471 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd"] Oct 31 00:32:29.699253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699219 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699538 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdv7j\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699663 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699731 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699805 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699854 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699876 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699922 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.699986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.699955 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.801289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.801249 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdv7j\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.801604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.801581 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.801761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.801743 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.801918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.801900 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.802073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802058 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.802232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802213 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.802399 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802385 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.802480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802451 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.802637 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802508 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.802748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802732 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.803034 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.803015 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.803112 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.802542 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.803302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.803278 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.803457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.803339 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.806098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.806079 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.808050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.807870 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.872032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.871955 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdv7j\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.880741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.880672 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token\") pod \"e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:29.895536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:29.895507 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:30.082632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:30.082591 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:30.384408 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:32:30.384376 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd292ef_804f_4898_8e7b_0db768cd0930.slice/crio-705bf462a5864f5072b58bddf75b9d265571de8121d49b56f56680f91fa10e27 WatchSource:0}: Error finding container 705bf462a5864f5072b58bddf75b9d265571de8121d49b56f56680f91fa10e27: Status 404 returned error can't find the container with id 705bf462a5864f5072b58bddf75b9d265571de8121d49b56f56680f91fa10e27 Oct 31 00:32:30.387133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:30.387093 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:32:30.387290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:30.387190 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:32:30.387290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:30.387225 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:32:30.421523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:30.421489 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd"] Oct 31 00:32:30.429434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:30.429385 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:31.013158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:31.013121 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" event={"ID":"abd292ef-804f-4898-8e7b-0db768cd0930","Type":"ContainerStarted","Data":"046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6"} Oct 31 00:32:31.013158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:31.013204 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" event={"ID":"abd292ef-804f-4898-8e7b-0db768cd0930","Type":"ContainerStarted","Data":"705bf462a5864f5072b58bddf75b9d265571de8121d49b56f56680f91fa10e27"} Oct 31 00:32:31.135098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:31.135028 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podStartSLOduration=2.135007392 podStartE2EDuration="2.135007392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:31.132172513 +0000 UTC m=+1731.225462751" watchObservedRunningTime="2026-10-31 00:32:31.135007392 +0000 UTC m=+1731.228297579" Oct 31 00:32:31.902921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:31.902884 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:31.909040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:31.909011 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:32.021246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:32.021218 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:32.022287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:32.022265 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:32:34.015119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:34.015082 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:34.195393 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:34.195359 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:34.363951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:34.363860 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:35.396078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:35.396033 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:40.379659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:40.379625 5943 scope.go:122] "RemoveContainer" containerID="cd9285380b2a792e7e2a7805b52e13670c4805d5439bb56638be4756bbc77d1d" Oct 31 00:32:40.393021 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:40.393001 5943 scope.go:122] "RemoveContainer" containerID="f4b5b53a05a228f1983eb79326311f043f76e354d7ea1b78358fa96caaa1e8c6" Oct 31 00:32:40.402619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:40.402598 5943 scope.go:122] "RemoveContainer" containerID="c7fe53ffb31b53841e977e91e111b4267b0262a56f3e2bd49b9859314cf15f0b" Oct 31 00:32:40.410928 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:40.410901 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:40.865613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:40.865579 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:41.059122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:41.059082 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:41.248058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:41.247962 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:46.047304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:46.047270 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:47.905415 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:47.905376 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:51.372204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:51.372138 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:54.998842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:54.998803 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:55.442063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:55.442018 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6"] Oct 31 00:32:55.808682 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:55.808643 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.007055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:56.006990 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.162447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:56.162360 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.480336 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:56.480231 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.693875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:56.693382 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.930709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:56.930677 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:57.485374 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:32:57.485338 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:32:58.637528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:58.637501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:58.788147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:32:58.788113 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:00.443008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.442970 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" podUID="503141a3-2c97-4781-a631-5b74e280e75d" containerName="payload-processing" containerID="cri-o://ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b" gracePeriod=25 Oct 31 00:33:00.738727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.738697 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:33:00.802580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.802545 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh\" (UniqueName: \"kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh\") pod \"503141a3-2c97-4781-a631-5b74e280e75d\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " Oct 31 00:33:00.802830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.802812 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume\") pod \"503141a3-2c97-4781-a631-5b74e280e75d\" (UID: \"503141a3-2c97-4781-a631-5b74e280e75d\") " Oct 31 00:33:00.803462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.803257 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume" pod "503141a3-2c97-4781-a631-5b74e280e75d" (UID: "503141a3-2c97-4781-a631-5b74e280e75d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:00.803594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.803512 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503141a3-2c97-4781-a631-5b74e280e75d-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:00.804777 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.804755 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh" pod "503141a3-2c97-4781-a631-5b74e280e75d" (UID: "503141a3-2c97-4781-a631-5b74e280e75d"). InnerVolumeSpecName "kube-api-access-m24vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:00.904868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:00.904833 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m24vh\" (UniqueName: \"kubernetes.io/projected/503141a3-2c97-4781-a631-5b74e280e75d-kube-api-access-m24vh\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.220314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.220269 5943 generic.go:356] "Generic (PLEG): container finished" podID="503141a3-2c97-4781-a631-5b74e280e75d" containerID="ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b" exitCode=0 Oct 31 00:33:01.220485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.220323 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" Oct 31 00:33:01.220485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.220324 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" event={"ID":"503141a3-2c97-4781-a631-5b74e280e75d","Type":"ContainerDied","Data":"ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b"} Oct 31 00:33:01.220485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.220378 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6" event={"ID":"503141a3-2c97-4781-a631-5b74e280e75d","Type":"ContainerDied","Data":"e022499b375a477c87c28cd1bd649554aa4935cce5c7bd1cb2fda9a37a8b6e5f"} Oct 31 00:33:01.220485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.220407 5943 scope.go:122] "RemoveContainer" containerID="ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b" Oct 31 00:33:01.232070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.232050 5943 scope.go:122] "RemoveContainer" containerID="ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b" Oct 31 00:33:01.232391 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:33:01.232372 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b\": container with ID starting with ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b not found: ID does not exist" containerID="ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b" Oct 31 00:33:01.232516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.232401 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b"} err="failed to get container status \"ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b\": rpc error: code = NotFound desc = could not find container \"ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b\": container with ID starting with ed8b3648ba487db3fe488c9c2cd0c97c1bb9a75de893629cd364af23b585103b not found: ID does not exist" Oct 31 00:33:01.390339 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.390304 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6"] Oct 31 00:33:01.397027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.396999 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-835921-5495c6d7dc-gz5g6"] Oct 31 00:33:01.401302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.401277 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:01.552692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.552652 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:01.711898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:01.711862 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:02.042948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:02.042904 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503141a3-2c97-4781-a631-5b74e280e75d" path="/var/lib/kubelet/pods/503141a3-2c97-4781-a631-5b74e280e75d/volumes" Oct 31 00:33:02.540555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:02.540515 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:02.755244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:02.755205 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:02.921122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:02.921023 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:05.755912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:05.755870 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:05.953150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:05.953110 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:06.949994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:06.949952 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:09.734852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:09.734816 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:09.920569 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:09.920525 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:10.830353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:10.830312 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:11.408899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:11.408853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:13.031702 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:13.031665 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:13.298585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:13.298468 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:13.453732 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:13.453702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:14.016228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:14.016156 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:18.516549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:18.516511 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:20.102744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:20.102702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:20.252516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:20.252480 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:20.476124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:20.476029 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:21.319408 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.319367 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc"] Oct 31 00:33:21.320246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.320226 5943 cpu_manager.go:414] "containerMap: removing container" podUID="503141a3-2c97-4781-a631-5b74e280e75d" containerName="payload-processing" Oct 31 00:33:21.320246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.320245 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503141a3-2c97-4781-a631-5b74e280e75d" containerName="payload-processing" Oct 31 00:33:21.320405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.320394 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="503141a3-2c97-4781-a631-5b74e280e75d" containerName="payload-processing" Oct 31 00:33:21.325115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.325095 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.381764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.381725 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc"] Oct 31 00:33:21.427787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.427750 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:33:21.427787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.427750 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:33:21.427787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.427752 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:33:21.428113 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.428095 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:33:21.431812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.431784 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.432050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.432029 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrj8\" (UniqueName: \"kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.432158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.432142 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.533941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.533903 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.534156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.534135 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrj8\" (UniqueName: \"kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.534258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.534241 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.534635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.534616 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.536901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.536868 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.594950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.594876 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrj8\" (UniqueName: \"kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8\") pod \"maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:21.641585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:21.641555 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:22.019519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:22.019487 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc"] Oct 31 00:33:22.021232 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:33:22.021211 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92938b6_b1c7_462e_8ece_1aa3d5fcde78.slice/crio-e0e7bf02c433b6be61f838da57e016d87a7e99775783290425aabdf80ad2ee0a WatchSource:0}: Error finding container e0e7bf02c433b6be61f838da57e016d87a7e99775783290425aabdf80ad2ee0a: Status 404 returned error can't find the container with id e0e7bf02c433b6be61f838da57e016d87a7e99775783290425aabdf80ad2ee0a Oct 31 00:33:22.365426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:22.365392 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" event={"ID":"f92938b6-b1c7-462e-8ece-1aa3d5fcde78","Type":"ContainerStarted","Data":"966f14f94eca4b39e4361d61595a76404431291cba458831bba965513089a3b9"} Oct 31 00:33:22.365805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:22.365453 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" event={"ID":"f92938b6-b1c7-462e-8ece-1aa3d5fcde78","Type":"ContainerStarted","Data":"e0e7bf02c433b6be61f838da57e016d87a7e99775783290425aabdf80ad2ee0a"} Oct 31 00:33:22.482776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:22.482716 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" podStartSLOduration=1.482701346 podStartE2EDuration="1.482701346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:22.480135595 +0000 UTC m=+1782.573425818" watchObservedRunningTime="2026-10-31 00:33:22.482701346 +0000 UTC m=+1782.575991533" Oct 31 00:33:23.372907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:23.372867 5943 generic.go:356] "Generic (PLEG): container finished" podID="f92938b6-b1c7-462e-8ece-1aa3d5fcde78" containerID="966f14f94eca4b39e4361d61595a76404431291cba458831bba965513089a3b9" exitCode=0 Oct 31 00:33:23.373310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:23.372933 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" event={"ID":"f92938b6-b1c7-462e-8ece-1aa3d5fcde78","Type":"ContainerDied","Data":"966f14f94eca4b39e4361d61595a76404431291cba458831bba965513089a3b9"} Oct 31 00:33:23.722233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:23.722138 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:23.844064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:23.844031 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:33:24.533147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.533122 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:24.686580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.685409 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca\") pod \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " Oct 31 00:33:24.686580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.685529 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8\" (UniqueName: \"kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8\") pod \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " Oct 31 00:33:24.686580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.685686 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca\") pod \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\" (UID: \"f92938b6-b1c7-462e-8ece-1aa3d5fcde78\") " Oct 31 00:33:24.687306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.687038 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca" pod "f92938b6-b1c7-462e-8ece-1aa3d5fcde78" (UID: "f92938b6-b1c7-462e-8ece-1aa3d5fcde78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:24.690833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.690647 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8" pod "f92938b6-b1c7-462e-8ece-1aa3d5fcde78" (UID: "f92938b6-b1c7-462e-8ece-1aa3d5fcde78"). InnerVolumeSpecName "kube-api-access-xfrj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:24.690833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.690775 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca" pod "f92938b6-b1c7-462e-8ece-1aa3d5fcde78" (UID: "f92938b6-b1c7-462e-8ece-1aa3d5fcde78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:24.787900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.787863 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrj8\" (UniqueName: \"kubernetes.io/projected/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-kube-api-access-xfrj8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:24.787900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.787900 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:24.788124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:24.787914 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92938b6-b1c7-462e-8ece-1aa3d5fcde78-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:25.386733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:25.386704 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" Oct 31 00:33:25.386924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:25.386748 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc" event={"ID":"f92938b6-b1c7-462e-8ece-1aa3d5fcde78","Type":"ContainerDied","Data":"e0e7bf02c433b6be61f838da57e016d87a7e99775783290425aabdf80ad2ee0a"} Oct 31 00:33:25.386924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:25.386809 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e7bf02c433b6be61f838da57e016d87a7e99775783290425aabdf80ad2ee0a" Oct 31 00:33:27.486981 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:33:27.486940 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:33:28.981812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:28.981770 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.898713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:32.898672 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq"] Oct 31 00:33:33.152670 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:33.152580 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:33.345317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:33.345279 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:33.964987 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:33.964946 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:34.128378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:34.128336 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:34.274923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:34.274833 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:37.900006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:37.899929 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" podUID="42f1ce65-d6e0-48bd-a53c-857d2f73239f" containerName="payload-pre-processing" containerID="cri-o://abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96" gracePeriod=25 Oct 31 00:33:38.189466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.189442 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:33:38.306526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.306496 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw\" (UniqueName: \"kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw\") pod \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " Oct 31 00:33:38.306694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.306547 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume\") pod \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\" (UID: \"42f1ce65-d6e0-48bd-a53c-857d2f73239f\") " Oct 31 00:33:38.306951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.306925 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume" pod "42f1ce65-d6e0-48bd-a53c-857d2f73239f" (UID: "42f1ce65-d6e0-48bd-a53c-857d2f73239f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:38.308840 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.308820 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw" pod "42f1ce65-d6e0-48bd-a53c-857d2f73239f" (UID: "42f1ce65-d6e0-48bd-a53c-857d2f73239f"). InnerVolumeSpecName "kube-api-access-4fthw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:38.408032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.407993 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fthw\" (UniqueName: \"kubernetes.io/projected/42f1ce65-d6e0-48bd-a53c-857d2f73239f-kube-api-access-4fthw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:38.408032 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.408026 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f1ce65-d6e0-48bd-a53c-857d2f73239f-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:38.477233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.477130 5943 generic.go:356] "Generic (PLEG): container finished" podID="42f1ce65-d6e0-48bd-a53c-857d2f73239f" containerID="abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96" exitCode=0 Oct 31 00:33:38.477233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.477221 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" Oct 31 00:33:38.477233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.477223 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" event={"ID":"42f1ce65-d6e0-48bd-a53c-857d2f73239f","Type":"ContainerDied","Data":"abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96"} Oct 31 00:33:38.477449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.477279 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq" event={"ID":"42f1ce65-d6e0-48bd-a53c-857d2f73239f","Type":"ContainerDied","Data":"069105150b17b6c0a79d33b020b76a7964f49ba3fc0c9e6ba22a6ffd4254bb56"} Oct 31 00:33:38.477449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.477309 5943 scope.go:122] "RemoveContainer" containerID="abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96" Oct 31 00:33:38.488473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.488456 5943 scope.go:122] "RemoveContainer" containerID="abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96" Oct 31 00:33:38.488758 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:33:38.488738 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96\": container with ID starting with abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96 not found: ID does not exist" containerID="abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96" Oct 31 00:33:38.488823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.488774 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96"} err="failed to get container status \"abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96\": rpc error: code = NotFound desc = could not find container \"abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96\": container with ID starting with abf22539703bbec3956c23838a8dd68f5fd3e4cc53c72b60506e23f0804afe96 not found: ID does not exist" Oct 31 00:33:38.651173 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.651138 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq"] Oct 31 00:33:38.657613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:38.657587 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ccdde-65dcc85c9-254dq"] Oct 31 00:33:39.404896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:39.404864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:40.041938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:40.041909 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f1ce65-d6e0-48bd-a53c-857d2f73239f" path="/var/lib/kubelet/pods/42f1ce65-d6e0-48bd-a53c-857d2f73239f/volumes" Oct 31 00:33:40.435956 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:40.435871 5943 scope.go:122] "RemoveContainer" containerID="b859818ffd19eb7af2b5c41b2356b6c9e9686de5d1385fa95702ded0ec9062fc" Oct 31 00:33:44.246674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:44.246494 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:45.772528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.772495 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq"] Oct 31 00:33:45.773266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.773249 5943 cpu_manager.go:414] "containerMap: removing container" podUID="42f1ce65-d6e0-48bd-a53c-857d2f73239f" containerName="payload-pre-processing" Oct 31 00:33:45.773340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.773267 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f1ce65-d6e0-48bd-a53c-857d2f73239f" containerName="payload-pre-processing" Oct 31 00:33:45.773340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.773298 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f92938b6-b1c7-462e-8ece-1aa3d5fcde78" containerName="revoke-keys" Oct 31 00:33:45.773340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.773304 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92938b6-b1c7-462e-8ece-1aa3d5fcde78" containerName="revoke-keys" Oct 31 00:33:45.773440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.773413 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f1ce65-d6e0-48bd-a53c-857d2f73239f" containerName="payload-pre-processing" Oct 31 00:33:45.773440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.773425 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92938b6-b1c7-462e-8ece-1aa3d5fcde78" containerName="revoke-keys" Oct 31 00:33:45.778305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.778286 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.832748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.832717 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7c92a85-openshift-default-dockercfg-rt7pg\"" Oct 31 00:33:45.886246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.886214 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq"] Oct 31 00:33:45.913656 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.913623 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.913835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.913723 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.913934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.913906 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.914023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.914002 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.914093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.914075 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.914149 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.914117 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.914231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.914156 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzqm\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.914427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.914401 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:45.914478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:45.914453 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015771 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015737 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015790 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015855 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015888 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015910 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015933 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.015975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.015954 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzqm\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016066 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016206 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016391 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016555 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016595 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016627 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.016848 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.016825 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.019040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.019018 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.019209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.019169 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.077632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.077557 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.077758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.077693 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzqm\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.092838 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.092811 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:46.536647 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:33:46.536618 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae77f639_1cd8_4cb6_b543_736d30b20235.slice/crio-70639acb6784ba016e17da01798a9ecc34f2376ade4e7c199cbdb67b630ea4fb WatchSource:0}: Error finding container 70639acb6784ba016e17da01798a9ecc34f2376ade4e7c199cbdb67b630ea4fb: Status 404 returned error can't find the container with id 70639acb6784ba016e17da01798a9ecc34f2376ade4e7c199cbdb67b630ea4fb Oct 31 00:33:46.537712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.537687 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq"] Oct 31 00:33:46.539146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.539115 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:33:46.539262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.539203 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:33:46.539262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:46.539234 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:33:47.559795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:47.559748 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" event={"ID":"ae77f639-1cd8-4cb6-b543-736d30b20235","Type":"ContainerStarted","Data":"3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183"} Oct 31 00:33:47.560253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:47.559858 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" event={"ID":"ae77f639-1cd8-4cb6-b543-736d30b20235","Type":"ContainerStarted","Data":"70639acb6784ba016e17da01798a9ecc34f2376ade4e7c199cbdb67b630ea4fb"} Oct 31 00:33:47.684073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:47.684014 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podStartSLOduration=2.68399859 podStartE2EDuration="2.68399859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:47.680284194 +0000 UTC m=+1807.773574383" watchObservedRunningTime="2026-10-31 00:33:47.68399859 +0000 UTC m=+1807.777288777" Oct 31 00:33:48.093343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:48.093306 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:48.094989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:48.094968 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:33:48.095050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:48.095021 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:33:48.254308 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:48.254268 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.093606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:49.093348 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:33:49.093606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:49.093443 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:33:49.455690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:49.455592 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:50.093628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:50.093598 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:33:50.094023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:50.093661 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:33:51.026045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:51.024784 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs"] Oct 31 00:33:51.026045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:51.025140 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" containerID="cri-o://76e287a6dc546d15fd6a8d98ddf4f8df6a78a2c542cec55eff5b417ce896a05b" gracePeriod=30 Oct 31 00:33:51.093604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:51.093572 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:33:51.093800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:51.093644 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:33:52.093404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:52.093368 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:33:52.093620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:52.093430 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:33:53.008387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:53.008355 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:53.129854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:53.129822 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:53.130354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:53.130302 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:53.131400 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:53.131377 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:33:53.500636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:53.500542 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:55.000824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:55.000788 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:33:55.960822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:55.960774 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:33:55.961005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:55.960844 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:33:56.152192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.152117 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:56.623645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.623606 5943 generic.go:356] "Generic (PLEG): container finished" podID="212539d4-2703-4468-932e-66b898149beb" containerID="76e287a6dc546d15fd6a8d98ddf4f8df6a78a2c542cec55eff5b417ce896a05b" exitCode=0 Oct 31 00:33:56.623824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.623706 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" event={"ID":"212539d4-2703-4468-932e-66b898149beb","Type":"ContainerDied","Data":"76e287a6dc546d15fd6a8d98ddf4f8df6a78a2c542cec55eff5b417ce896a05b"} Oct 31 00:33:56.780370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.780344 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:33:56.910611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910528 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.910763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910672 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.910763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910696 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.910763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910759 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.910890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910787 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.910890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910870 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.910995 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910963 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.911054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.910997 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.911054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911018 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs\") pod \"212539d4-2703-4468-932e-66b898149beb\" (UID: \"212539d4-2703-4468-932e-66b898149beb\") " Oct 31 00:33:56.911158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911077 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:56.911158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911091 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.911345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911295 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.911410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911340 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.911484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911456 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.911935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911829 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212539d4-2703-4468-932e-66b898149beb-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.911935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911857 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.911935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911875 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.911935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911891 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.911935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.911908 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.913113 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.913088 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "kube-api-access-qvmq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:56.913242 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.913113 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:56.913308 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.913269 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.913482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:56.913461 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo" pod "212539d4-2703-4468-932e-66b898149beb" (UID: "212539d4-2703-4468-932e-66b898149beb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:57.012820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.012783 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmq6\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-kube-api-access-qvmq6\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:57.012820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.012818 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212539d4-2703-4468-932e-66b898149beb-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:57.012820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.012832 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212539d4-2703-4468-932e-66b898149beb-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:57.013064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.012845 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212539d4-2703-4468-932e-66b898149beb-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:57.485803 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:33:57.485767 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:33:57.657704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.657674 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" Oct 31 00:33:57.657921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.657703 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs" event={"ID":"212539d4-2703-4468-932e-66b898149beb","Type":"ContainerDied","Data":"2a77dccf42088a0366e9bfa05bb101380a8f0d41393de6ec7b06170529824d09"} Oct 31 00:33:57.657921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.657781 5943 scope.go:122] "RemoveContainer" containerID="76e287a6dc546d15fd6a8d98ddf4f8df6a78a2c542cec55eff5b417ce896a05b" Oct 31 00:33:57.872268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.872233 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs"] Oct 31 00:33:57.922108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:57.922081 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-j67vs"] Oct 31 00:33:58.061998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:58.061962 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212539d4-2703-4468-932e-66b898149beb" path="/var/lib/kubelet/pods/212539d4-2703-4468-932e-66b898149beb/volumes" Oct 31 00:33:58.062330 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:58.062316 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m"] Oct 31 00:33:58.062378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:33:58.062348 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2811c3-kbr4m"] Oct 31 00:34:00.060230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.060196 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867a76a9-5d03-4fd2-9d63-8461aa340bc1" path="/var/lib/kubelet/pods/867a76a9-5d03-4fd2-9d63-8461aa340bc1/volumes" Oct 31 00:34:00.913865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.913833 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l"] Oct 31 00:34:00.914516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.914500 5943 cpu_manager.go:414] "containerMap: removing container" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" Oct 31 00:34:00.914586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.914518 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" Oct 31 00:34:00.914647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.914635 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="212539d4-2703-4468-932e-66b898149beb" containerName="istio-proxy" Oct 31 00:34:00.918190 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.918155 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:00.954295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:00.954249 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:01.088302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.088273 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l"] Oct 31 00:34:01.098657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.098621 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.098934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.098918 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099106 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099166 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099241 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099494 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099473 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099582 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099534 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ck8\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099595 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.099741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.099720 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201386 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ck8\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201427 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201458 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201552 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201594 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201883 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201800 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201876 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.201950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201924 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.202061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.201952 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.202235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.202213 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.202308 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.202282 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.202529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.202513 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.202607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.202584 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.202886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.202860 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.204222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.204168 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.204331 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.204315 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.262745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.262708 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.262918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.262892 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ck8\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.554869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.554832 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:01.994580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.994549 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l"] Oct 31 00:34:01.995280 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:34:01.995250 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ddf82f_c378_4f70_9352_a820f0dc0b12.slice/crio-f47ff7164ec7e96e8e815c21b3a1ffc0ed448ce85bd08f36922e09de2b63b758 WatchSource:0}: Error finding container f47ff7164ec7e96e8e815c21b3a1ffc0ed448ce85bd08f36922e09de2b63b758: Status 404 returned error can't find the container with id f47ff7164ec7e96e8e815c21b3a1ffc0ed448ce85bd08f36922e09de2b63b758 Oct 31 00:34:01.998065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.998036 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:01.998137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.998099 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:01.998137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:01.998130 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:02.698139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:02.698101 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" event={"ID":"a3ddf82f-c378-4f70-9352-a820f0dc0b12","Type":"ContainerStarted","Data":"d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa"} Oct 31 00:34:02.698684 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:02.698161 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" event={"ID":"a3ddf82f-c378-4f70-9352-a820f0dc0b12","Type":"ContainerStarted","Data":"f47ff7164ec7e96e8e815c21b3a1ffc0ed448ce85bd08f36922e09de2b63b758"} Oct 31 00:34:02.923958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:02.923904 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podStartSLOduration=2.923888197 podStartE2EDuration="2.923888197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.872845437 +0000 UTC m=+1822.966135641" watchObservedRunningTime="2026-10-31 00:34:02.923888197 +0000 UTC m=+1823.017178439" Oct 31 00:34:03.063442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:03.063392 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:34:03.063635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:03.063477 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:34:03.556006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:03.555967 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:03.557572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:03.557539 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:34:03.557704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:03.557608 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:34:04.555834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:04.555793 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:34:04.556250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:04.555861 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:34:05.555751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:05.555710 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:34:05.555939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:05.555771 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:34:05.750203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:05.728612 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:05.886436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:05.886330 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:06.084669 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.084633 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:06.556605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.556568 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:34:06.557064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.556636 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:34:06.727946 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.727915 5943 generic.go:356] "Generic (PLEG): container finished" podID="abd292ef-804f-4898-8e7b-0db768cd0930" containerID="046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6" exitCode=0 Oct 31 00:34:06.728117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.727990 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" event={"ID":"abd292ef-804f-4898-8e7b-0db768cd0930","Type":"ContainerDied","Data":"046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6"} Oct 31 00:34:06.739207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.728553 5943 scope.go:122] "RemoveContainer" containerID="046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6" Oct 31 00:34:06.749202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.743777 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:06.749500 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.749468 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:06.749653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:06.749634 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:07.555669 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:07.555612 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:34:07.555891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:07.555683 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:34:07.772203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:07.771648 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" event={"ID":"abd292ef-804f-4898-8e7b-0db768cd0930","Type":"ContainerStarted","Data":"8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3"} Oct 31 00:34:07.903263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:07.902924 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:34:07.912226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:07.910292 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:34:07.912226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:07.910351 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:34:08.649206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:08.644523 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:08.659285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:08.649889 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:08.659285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:08.658687 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:34:08.904010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:08.903745 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:34:08.904010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:08.903817 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:34:09.902778 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:09.902748 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:34:09.902959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:09.902812 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:34:10.827578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:10.827543 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:10.902892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:10.902846 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:34:10.903067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:10.902931 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:34:11.902746 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:11.902703 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:34:11.903274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:11.902777 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:34:12.903203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:12.903141 5943 patch_prober.go:29] interesting pod/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:34:12.903614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:12.903243 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:34:13.908039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:13.908004 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:34:13.908521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:13.908467 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:34:13.909151 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:13.909135 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:34:15.761714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:15.761672 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:16.023646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.023558 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t"] Oct 31 00:34:16.027712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.027689 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.119117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.119049 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.119425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.119399 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.119576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.119546 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.119796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.119773 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.119939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.119921 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.119984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.119957 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.120096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.120070 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.120135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.120119 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.120263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.120249 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.133474 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.133445 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t"] Oct 31 00:34:16.222053 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222015 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222053 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222055 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222103 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222136 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222164 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222351 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222560 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222472 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222560 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222506 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222560 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222554 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222586 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222766 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222826 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.222974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222951 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.223019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.222985 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.224798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.224773 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.225059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.225041 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.284475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.284387 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.285035 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.285004 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.342150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.342122 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:16.773588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.773562 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t"] Oct 31 00:34:16.775233 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:34:16.775204 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5190d3_3e4b_4ed1_b72e_755df918cb31.slice/crio-11dff64f6d316e2bc45a0793e7cd604bbe95ae51b01ad5748c6e870cdf1d282d WatchSource:0}: Error finding container 11dff64f6d316e2bc45a0793e7cd604bbe95ae51b01ad5748c6e870cdf1d282d: Status 404 returned error can't find the container with id 11dff64f6d316e2bc45a0793e7cd604bbe95ae51b01ad5748c6e870cdf1d282d Oct 31 00:34:16.777789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.777754 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:16.777895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.777836 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:16.777895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.777882 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:34:16.837663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:16.837632 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" event={"ID":"ee5190d3-3e4b-4ed1-b72e-755df918cb31","Type":"ContainerStarted","Data":"11dff64f6d316e2bc45a0793e7cd604bbe95ae51b01ad5748c6e870cdf1d282d"} Oct 31 00:34:17.860189 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:17.860146 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" event={"ID":"ee5190d3-3e4b-4ed1-b72e-755df918cb31","Type":"ContainerStarted","Data":"8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3"} Oct 31 00:34:18.057882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:18.057804 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podStartSLOduration=3.0577823 podStartE2EDuration="3.0577823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:18.055450963 +0000 UTC m=+1838.148741151" watchObservedRunningTime="2026-10-31 00:34:18.0577823 +0000 UTC m=+1838.151072512" Oct 31 00:34:18.350204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:18.349525 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:18.353332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:18.353298 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:34:18.353578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:18.353543 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:34:19.346636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:19.346596 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:34:19.348026 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:19.347990 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:34:20.303334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:20.302786 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:20.343287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:20.343255 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:34:20.343608 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:20.343562 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:34:21.347231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:21.347039 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:34:21.347231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:21.347111 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:34:21.915591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:21.915552 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:22.156450 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:22.156005 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:22.311054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:22.311017 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:22.347637 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:22.347599 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:34:22.348334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:22.348296 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:34:22.907511 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:22.907465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:23.366893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:23.366864 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:23.367646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:23.367302 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:23.368328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:23.368306 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:34:26.977633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:26.977595 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6"] Oct 31 00:34:26.982851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:26.982833 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.086392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.086362 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:34:27.086392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.086387 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:34:27.086619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.086411 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:34:27.086619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.086529 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:34:27.089737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.089718 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6"] Oct 31 00:34:27.137385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.137349 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkzg\" (UniqueName: \"kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.137636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.137606 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.137800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.137784 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.239304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.239218 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.239488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.239342 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkzg\" (UniqueName: \"kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.239488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.239464 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.239960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.239935 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.241989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.241973 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.301549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.301517 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkzg\" (UniqueName: \"kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.331527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.331489 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:27.476466 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:34:27.476424 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:34:27.597462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.597426 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:27.974505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:27.974469 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6"] Oct 31 00:34:27.975774 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:34:27.975747 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8caa8d3_1a3f_4c72_ae34_5ea44a074973.slice/crio-81979947790fe1a227c60c334d3262d9262bff6aefe5dd186c3f75b289b4de63 WatchSource:0}: Error finding container 81979947790fe1a227c60c334d3262d9262bff6aefe5dd186c3f75b289b4de63: Status 404 returned error can't find the container with id 81979947790fe1a227c60c334d3262d9262bff6aefe5dd186c3f75b289b4de63 Oct 31 00:34:28.772402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:28.772370 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:28.934883 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:28.934852 5943 generic.go:356] "Generic (PLEG): container finished" podID="f8caa8d3-1a3f-4c72-ae34-5ea44a074973" containerID="e8c057d953fe32d97320ee431edd487a0e32742874681e86e5858b72da825f04" exitCode=0 Oct 31 00:34:28.935061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:28.934935 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" event={"ID":"f8caa8d3-1a3f-4c72-ae34-5ea44a074973","Type":"ContainerDied","Data":"e8c057d953fe32d97320ee431edd487a0e32742874681e86e5858b72da825f04"} Oct 31 00:34:28.935061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:28.934996 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" event={"ID":"f8caa8d3-1a3f-4c72-ae34-5ea44a074973","Type":"ContainerStarted","Data":"81979947790fe1a227c60c334d3262d9262bff6aefe5dd186c3f75b289b4de63"} Oct 31 00:34:28.966998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:28.966965 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:29.126325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:29.126220 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:29.943135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:29.943112 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds"] Oct 31 00:34:29.947201 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:29.947169 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.012144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.012106 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds"] Oct 31 00:34:30.093699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.093665 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.093833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.093766 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.093833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.093807 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfghz\" (UniqueName: \"kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.110564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.110541 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:30.194985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.194891 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.194985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.194970 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.195260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.195171 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfghz\" (UniqueName: \"kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.195660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.195629 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.197558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.197535 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.256171 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.256139 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfghz\" (UniqueName: \"kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz\") pod \"maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.261033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.261005 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:30.296677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.296633 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg\" (UniqueName: \"kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg\") pod \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " Oct 31 00:34:30.296677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.296673 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca\") pod \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " Oct 31 00:34:30.296929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.296880 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca\") pod \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\" (UID: \"f8caa8d3-1a3f-4c72-ae34-5ea44a074973\") " Oct 31 00:34:30.297134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.297111 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca" pod "f8caa8d3-1a3f-4c72-ae34-5ea44a074973" (UID: "f8caa8d3-1a3f-4c72-ae34-5ea44a074973"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:30.297803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.297779 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:30.299029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.299004 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg" pod "f8caa8d3-1a3f-4c72-ae34-5ea44a074973" (UID: "f8caa8d3-1a3f-4c72-ae34-5ea44a074973"). InnerVolumeSpecName "kube-api-access-9mkzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:30.299352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.299331 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca" pod "f8caa8d3-1a3f-4c72-ae34-5ea44a074973" (UID: "f8caa8d3-1a3f-4c72-ae34-5ea44a074973"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:30.399022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.398990 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mkzg\" (UniqueName: \"kubernetes.io/projected/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-kube-api-access-9mkzg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:30.399022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.399021 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8caa8d3-1a3f-4c72-ae34-5ea44a074973-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:30.651393 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.651367 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds"] Oct 31 00:34:30.652273 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:34:30.652250 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1621cbb8_1ba0_4b8f_9740_cff0f15f11a2.slice/crio-cfc24d4c7c0f6c0b74c6e9329a806c88f3991fc26e1737d9d0e6d82907937afa WatchSource:0}: Error finding container cfc24d4c7c0f6c0b74c6e9329a806c88f3991fc26e1737d9d0e6d82907937afa: Status 404 returned error can't find the container with id cfc24d4c7c0f6c0b74c6e9329a806c88f3991fc26e1737d9d0e6d82907937afa Oct 31 00:34:30.950547 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.950513 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" event={"ID":"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2","Type":"ContainerStarted","Data":"60dead7e78ba60393cd5e376c3dcc536cc11f0fcc508aaed94148588c4942304"} Oct 31 00:34:30.950998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.950577 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" event={"ID":"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2","Type":"ContainerStarted","Data":"cfc24d4c7c0f6c0b74c6e9329a806c88f3991fc26e1737d9d0e6d82907937afa"} Oct 31 00:34:30.951936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.951916 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" Oct 31 00:34:30.952057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.951914 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6" event={"ID":"f8caa8d3-1a3f-4c72-ae34-5ea44a074973","Type":"ContainerDied","Data":"81979947790fe1a227c60c334d3262d9262bff6aefe5dd186c3f75b289b4de63"} Oct 31 00:34:30.952057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:30.952042 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81979947790fe1a227c60c334d3262d9262bff6aefe5dd186c3f75b289b4de63" Oct 31 00:34:31.068772 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:31.068717 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" podStartSLOduration=2.068698461 podStartE2EDuration="2.068698461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:31.065746249 +0000 UTC m=+1851.159036455" watchObservedRunningTime="2026-10-31 00:34:31.068698461 +0000 UTC m=+1851.161988654" Oct 31 00:34:31.960134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:31.960050 5943 generic.go:356] "Generic (PLEG): container finished" podID="1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" containerID="60dead7e78ba60393cd5e376c3dcc536cc11f0fcc508aaed94148588c4942304" exitCode=0 Oct 31 00:34:31.960516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:31.960138 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" event={"ID":"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2","Type":"ContainerDied","Data":"60dead7e78ba60393cd5e376c3dcc536cc11f0fcc508aaed94148588c4942304"} Oct 31 00:34:32.653290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:32.653243 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:33.124338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.124311 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:33.145578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.145546 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz\" (UniqueName: \"kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz\") pod \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " Oct 31 00:34:33.145749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.145594 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca\") pod \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " Oct 31 00:34:33.145749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.145662 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca\") pod \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\" (UID: \"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2\") " Oct 31 00:34:33.146144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.146082 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca" pod "1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" (UID: "1621cbb8-1ba0-4b8f-9740-cff0f15f11a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:33.146401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.146381 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:33.147922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.147897 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca" pod "1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" (UID: "1621cbb8-1ba0-4b8f-9740-cff0f15f11a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:33.148211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.148149 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz" pod "1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" (UID: "1621cbb8-1ba0-4b8f-9740-cff0f15f11a2"). InnerVolumeSpecName "kube-api-access-tfghz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:33.247699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.247612 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfghz\" (UniqueName: \"kubernetes.io/projected/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-kube-api-access-tfghz\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:33.247699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.247641 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:33.974008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.973973 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" Oct 31 00:34:33.974008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.973999 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds" event={"ID":"1621cbb8-1ba0-4b8f-9740-cff0f15f11a2","Type":"ContainerDied","Data":"cfc24d4c7c0f6c0b74c6e9329a806c88f3991fc26e1737d9d0e6d82907937afa"} Oct 31 00:34:33.974233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:33.974043 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfc24d4c7c0f6c0b74c6e9329a806c88f3991fc26e1737d9d0e6d82907937afa" Oct 31 00:34:35.183988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:35.183953 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:34:37.828965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:37.828926 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:38.907120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:38.907080 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:39.056744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:39.056688 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:41.445270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:41.445230 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:41.591319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:41.591282 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:43.341588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:43.341537 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:47.285913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:47.285852 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:48.535490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:48.535444 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:48.746050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:48.746009 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:49.009123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009091 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5"] Oct 31 00:34:49.009809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009789 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" containerName="revoke-keys" Oct 31 00:34:49.009863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009811 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" containerName="revoke-keys" Oct 31 00:34:49.009902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009868 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f8caa8d3-1a3f-4c72-ae34-5ea44a074973" containerName="revoke-keys" Oct 31 00:34:49.009902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009873 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8caa8d3-1a3f-4c72-ae34-5ea44a074973" containerName="revoke-keys" Oct 31 00:34:49.009981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009969 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8caa8d3-1a3f-4c72-ae34-5ea44a074973" containerName="revoke-keys" Oct 31 00:34:49.009981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.009978 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" containerName="revoke-keys" Oct 31 00:34:49.014788 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.014768 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.068629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.068591 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:34:49.118332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.118294 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:34:49.118332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.118299 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:34:49.118538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.118299 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:34:49.121509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.121488 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5"] Oct 31 00:34:49.205705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.205664 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xst\" (UniqueName: \"kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.205897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.205877 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.205954 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.205936 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.307305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.307169 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.307305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.307258 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.307514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.307443 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xst\" (UniqueName: \"kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.308104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.308078 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.309779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.309754 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.368620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.368585 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xst\" (UniqueName: \"kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst\") pod \"maas-api-revoke-keys-e2e-api-a-3e7872-hhft5\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:49.629062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:49.628977 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:50.008010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:50.007986 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5"] Oct 31 00:34:50.009772 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:34:50.009743 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dec3283_9ed0_4b54_84cf_fbe375295292.slice/crio-59321c727aeacc99e4cd9217a773d98e95f027ef0d619aff0d7ce383d4204ba3 WatchSource:0}: Error finding container 59321c727aeacc99e4cd9217a773d98e95f027ef0d619aff0d7ce383d4204ba3: Status 404 returned error can't find the container with id 59321c727aeacc99e4cd9217a773d98e95f027ef0d619aff0d7ce383d4204ba3 Oct 31 00:34:50.093416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:50.093378 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" event={"ID":"3dec3283-9ed0-4b54-84cf-fbe375295292","Type":"ContainerStarted","Data":"59321c727aeacc99e4cd9217a773d98e95f027ef0d619aff0d7ce383d4204ba3"} Oct 31 00:34:51.100975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:51.100935 5943 generic.go:356] "Generic (PLEG): container finished" podID="3dec3283-9ed0-4b54-84cf-fbe375295292" containerID="9262fcf892ac1e0d1fa80623666f79e1fe846fbdd3c8efda98ff6abd4a3c2946" exitCode=0 Oct 31 00:34:51.101461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:51.101076 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" event={"ID":"3dec3283-9ed0-4b54-84cf-fbe375295292","Type":"ContainerDied","Data":"9262fcf892ac1e0d1fa80623666f79e1fe846fbdd3c8efda98ff6abd4a3c2946"} Oct 31 00:34:52.276067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.276039 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:52.344996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.344966 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca\") pod \"3dec3283-9ed0-4b54-84cf-fbe375295292\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " Oct 31 00:34:52.345203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.345148 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst\" (UniqueName: \"kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst\") pod \"3dec3283-9ed0-4b54-84cf-fbe375295292\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " Oct 31 00:34:52.345425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.345405 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca\") pod \"3dec3283-9ed0-4b54-84cf-fbe375295292\" (UID: \"3dec3283-9ed0-4b54-84cf-fbe375295292\") " Oct 31 00:34:52.345782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.345760 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca" pod "3dec3283-9ed0-4b54-84cf-fbe375295292" (UID: "3dec3283-9ed0-4b54-84cf-fbe375295292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:52.345990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.345975 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.347250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.347230 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca" pod "3dec3283-9ed0-4b54-84cf-fbe375295292" (UID: "3dec3283-9ed0-4b54-84cf-fbe375295292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:52.347389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.347373 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst" pod "3dec3283-9ed0-4b54-84cf-fbe375295292" (UID: "3dec3283-9ed0-4b54-84cf-fbe375295292"). InnerVolumeSpecName "kube-api-access-t2xst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:52.447595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.447518 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dec3283-9ed0-4b54-84cf-fbe375295292-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.447595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:52.447550 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2xst\" (UniqueName: \"kubernetes.io/projected/3dec3283-9ed0-4b54-84cf-fbe375295292-kube-api-access-t2xst\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:53.116596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:53.116561 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" Oct 31 00:34:53.116596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:53.116577 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5" event={"ID":"3dec3283-9ed0-4b54-84cf-fbe375295292","Type":"ContainerDied","Data":"59321c727aeacc99e4cd9217a773d98e95f027ef0d619aff0d7ce383d4204ba3"} Oct 31 00:34:53.116805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:53.116627 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59321c727aeacc99e4cd9217a773d98e95f027ef0d619aff0d7ce383d4204ba3" Oct 31 00:34:53.977152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:34:53.977118 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:57.477622 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:34:57.477580 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:35:00.650380 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:00.650326 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:00.886737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:00.886695 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:01.030042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:01.030004 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:01.049364 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:01.049284 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2"] Oct 31 00:35:01.099148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:01.099117 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec38b57-9wrh2"] Oct 31 00:35:02.090392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:02.090336 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec" path="/var/lib/kubelet/pods/15b6fd36-64e0-4c13-858f-bf1cf0bbb5ec/volumes" Oct 31 00:35:05.480154 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:05.480115 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:10.258774 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:10.258735 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:14.409208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:14.408915 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:15.962896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:15.962851 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:16.290229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:16.290160 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:16.433477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:16.433448 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:16.572364 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:16.572284 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6"] Oct 31 00:35:16.621591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:16.621560 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2h4d6"] Oct 31 00:35:17.023096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:17.023060 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:18.042554 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:18.042511 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8caa8d3-1a3f-4c72-ae34-5ea44a074973" path="/var/lib/kubelet/pods/f8caa8d3-1a3f-4c72-ae34-5ea44a074973/volumes" Oct 31 00:35:20.188357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:20.188274 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:20.471229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:20.471107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:20.621958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:20.621883 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:21.645086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:21.645047 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:21.999623 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:21.999524 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g"] Oct 31 00:35:22.075131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:22.075098 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:25.854250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:25.854170 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:27.458696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:27.458660 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:27.477086 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:35:27.477045 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:35:27.740942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:27.740828 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:27.888410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:27.888361 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:27.960729 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:27.960684 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc"] Oct 31 00:35:27.963647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:27.963623 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ccdde-4jqdc"] Oct 31 00:35:28.042067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:28.042033 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92938b6-b1c7-462e-8ece-1aa3d5fcde78" path="/var/lib/kubelet/pods/f92938b6-b1c7-462e-8ece-1aa3d5fcde78/volumes" Oct 31 00:35:28.424821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:28.424725 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:32.889418 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:32.889380 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:34.588568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:34.588525 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:34.788608 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:34.788570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:34.975950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:34.975856 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:35.911519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:35.911487 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:36.779154 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:36.779118 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:37.018666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.018611 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:37.044884 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.044776 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" podUID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" containerName="main" containerID="cri-o://f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d" gracePeriod=45 Oct 31 00:35:37.342659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.342291 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:35:37.409715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.409684 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home\") pod \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " Oct 31 00:35:37.409923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.409797 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache\") pod \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " Oct 31 00:35:37.409923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.409903 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht\" (UniqueName: \"kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht\") pod \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " Oct 31 00:35:37.410049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.409968 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm\") pod \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " Oct 31 00:35:37.410105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410092 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs\") pod \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " Oct 31 00:35:37.410160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410129 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir\") pod \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\" (UID: \"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e\") " Oct 31 00:35:37.410527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410314 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home" pod "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" (UID: "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:37.410527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410341 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache" pod "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" (UID: "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:37.410703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410577 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir" pod "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" (UID: "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:37.410835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410815 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:37.410964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410838 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:37.410964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.410852 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:37.413104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.413080 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm" pod "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" (UID: "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:37.413236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.413205 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht" pod "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" (UID: "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e"). InnerVolumeSpecName "kube-api-access-2vrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:37.413284 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.413246 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs" pod "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" (UID: "a131f50e-c5e7-4f3f-a0d7-a89df8dd973e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:37.484405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.480853 5943 generic.go:356] "Generic (PLEG): container finished" podID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" containerID="f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d" exitCode=0 Oct 31 00:35:37.484405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.481091 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" event={"ID":"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e","Type":"ContainerDied","Data":"f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d"} Oct 31 00:35:37.484405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.481128 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" event={"ID":"a131f50e-c5e7-4f3f-a0d7-a89df8dd973e","Type":"ContainerDied","Data":"7a179ecdc7b1d32a1cdc4f348e84dfbc065fb9ddc7f36804e688d2787a694fdb"} Oct 31 00:35:37.484405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.481152 5943 scope.go:122] "RemoveContainer" containerID="f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d" Oct 31 00:35:37.484405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.481339 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" Oct 31 00:35:37.513157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.513083 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrht\" (UniqueName: \"kubernetes.io/projected/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-kube-api-access-2vrht\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:37.513157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.513118 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:37.513157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.513132 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:37.516301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.515753 5943 scope.go:122] "RemoveContainer" containerID="f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d" Oct 31 00:35:37.516301 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:35:37.516171 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d\": container with ID starting with f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d not found: ID does not exist" containerID="f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d" Oct 31 00:35:37.516301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.516226 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d"} err="failed to get container status \"f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d\": rpc error: code = NotFound desc = could not find container \"f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d\": container with ID starting with f63517ea2af79debf12e5d739e2915ed46ce19f9d14bed5963db89d47a02273d not found: ID does not exist" Oct 31 00:35:37.657766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.657675 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g"] Oct 31 00:35:37.665644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:37.665615 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g"] Oct 31 00:35:38.043785 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:38.043750 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" path="/var/lib/kubelet/pods/a131f50e-c5e7-4f3f-a0d7-a89df8dd973e/volumes" Oct 31 00:35:38.056294 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:38.056266 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:38.284354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:38.284313 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f7c92a85/e2e-lifecycle-model-f7c92a85-kserve-55cd8b984f-6sl8g" podUID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:35:40.522264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:40.522233 5943 scope.go:122] "RemoveContainer" containerID="3bc7ec587ed816e983420871e592a9f1893399b4c8343f990f04afa243685fd5" Oct 31 00:35:41.506010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:41.505972 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:41.750096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:41.750058 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:41.878352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:41.878284 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:35:43.323865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:43.323827 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:44.778040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:44.778005 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:45.101261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:45.095960 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:49.081169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:49.081138 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:49.959885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:49.959841 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:52.331077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:52.331034 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:52.476930 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:52.476893 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:54.752833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:54.752797 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:57.476502 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:35:57.476457 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:35:58.945336 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:58.945295 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:59.345665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:59.345629 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:59.917358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:35:59.917325 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:03.526520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:03.526482 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:03.993350 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:03.993256 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:04.967452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:04.967406 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf"] Oct 31 00:36:04.967920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:04.967665 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" containerID="cri-o://ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471" gracePeriod=30 Oct 31 00:36:05.582470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:05.582435 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:05.801094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:05.801053 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:05.978563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:05.978474 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq"] Oct 31 00:36:05.979063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:05.978749 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" containerID="cri-o://3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183" gracePeriod=30 Oct 31 00:36:05.984474 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:05.984443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:06.010839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.010795 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l"] Oct 31 00:36:06.011084 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.011046 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" containerID="cri-o://d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa" gracePeriod=30 Oct 31 00:36:06.015248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.015221 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t"] Oct 31 00:36:06.015561 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.015535 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" containerID="cri-o://8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3" gracePeriod=30 Oct 31 00:36:06.029962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.029934 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm"] Oct 31 00:36:06.030146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.030126 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" containerID="cri-o://386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991" gracePeriod=30 Oct 31 00:36:06.780986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.780937 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:06.975901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:06.975864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:07.028976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:07.028940 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw"] Oct 31 00:36:07.029429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:07.029143 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" containerID="cri-o://8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a" gracePeriod=30 Oct 31 00:36:07.151127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:07.151036 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2"] Oct 31 00:36:07.151323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:07.151284 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" containerID="cri-o://8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa" gracePeriod=30 Oct 31 00:36:08.092034 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.091991 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:08.133285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.133250 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.133445 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.133311 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:08.318046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.318008 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:08.369823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.369744 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.369823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.369799 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:08.652158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.652069 5943 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.652158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.652139 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:08.964827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.964732 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gltnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.964827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:08.964810 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:09.816918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:09.816879 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:09.817646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:09.817606 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:09.878352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:09.878311 5943 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:09.878533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:09.878382 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:10.236909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.236881 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:36:10.341355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341258 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341323 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341359 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341421 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341464 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341566 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341680 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341725 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341782 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data\") pod \"3218053c-1eb0-4bc8-9c6e-9654627540b2\" (UID: \"3218053c-1eb0-4bc8-9c6e-9654627540b2\") " Oct 31 00:36:10.341970 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341840 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:10.341970 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.341917 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:10.342382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342299 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:10.342382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342331 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:10.342747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342591 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.342747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342670 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.342747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342689 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3218053c-1eb0-4bc8-9c6e-9654627540b2-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.342747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342717 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.342747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.342667 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:10.344528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.344498 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:10.344528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.344498 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:10.344662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.344566 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "kube-api-access-d8ccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:10.344662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.344599 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo" pod "3218053c-1eb0-4bc8-9c6e-9654627540b2" (UID: "3218053c-1eb0-4bc8-9c6e-9654627540b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:10.443660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.443625 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.443660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.443655 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.443660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.443665 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ccs\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-kube-api-access-d8ccs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.444027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.443675 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.444027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.443683 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3218053c-1eb0-4bc8-9c6e-9654627540b2-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:10.716711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.716619 5943 generic.go:356] "Generic (PLEG): container finished" podID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerID="ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471" exitCode=0 Oct 31 00:36:10.716711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.716667 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" Oct 31 00:36:10.716920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.716707 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" event={"ID":"3218053c-1eb0-4bc8-9c6e-9654627540b2","Type":"ContainerDied","Data":"ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471"} Oct 31 00:36:10.716920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.716768 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf" event={"ID":"3218053c-1eb0-4bc8-9c6e-9654627540b2","Type":"ContainerDied","Data":"059ead77f1e02caef55a9afd2f7d71d5a63e4c6e7aed8266295c3a3aefdfacf5"} Oct 31 00:36:10.716920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.716798 5943 scope.go:122] "RemoveContainer" containerID="ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471" Oct 31 00:36:10.727807 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.727786 5943 scope.go:122] "RemoveContainer" containerID="ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471" Oct 31 00:36:10.728121 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:10.728100 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471\": container with ID starting with ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471 not found: ID does not exist" containerID="ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471" Oct 31 00:36:10.728254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.728135 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471"} err="failed to get container status \"ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471\": rpc error: code = NotFound desc = could not find container \"ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471\": container with ID starting with ba2ddf4a37c319607597a18e1f6b0c051ed07f977e08dfa4e1d2f2c8f9eba471 not found: ID does not exist" Oct 31 00:36:10.887287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.887251 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf"] Oct 31 00:36:10.936439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:10.936408 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-fj5mf"] Oct 31 00:36:11.334203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.331062 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:36:11.368259 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.368219 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:36:11.378355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.378327 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:36:11.422470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.422442 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:36:11.442523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.442491 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:11.442678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.442547 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:11.457937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.457902 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.458075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.457970 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.458075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458009 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.458075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458048 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.458279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458103 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.458279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458144 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.458279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458162 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.458279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458200 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.458479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458310 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.458479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458310 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.458479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458398 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.458479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458404 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.458479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458410 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458491 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458492 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458572 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458602 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458613 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458642 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.458736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458698 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458747 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458783 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458817 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458837 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458842 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458933 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.458987 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459120 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459200 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459224 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459263 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459315 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459349 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459382 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.459457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459439 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459475 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459534 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459576 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459623 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket\") pod \"ae77f639-1cd8-4cb6-b543-736d30b20235\" (UID: \"ae77f639-1cd8-4cb6-b543-736d30b20235\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459660 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459690 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459731 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert\") pod \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\" (UID: \"1b6c8d4d-73ba-4bb3-9b73-251785be67d3\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459778 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token\") pod \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\" (UID: \"ee5190d3-3e4b-4ed1-b72e-755df918cb31\") " Oct 31 00:36:11.459847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.459805 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo\") pod \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\" (UID: \"a3ddf82f-c378-4f70-9352-a820f0dc0b12\") " Oct 31 00:36:11.460532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460511 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460540 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460555 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460571 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460587 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460601 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460613 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460628 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.460988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460548 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.460988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460608 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.460988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460789 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.460988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460822 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.460988 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.460865 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.461369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.461072 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.461369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.461295 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.461472 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.461380 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.461689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.461410 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.461827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.461804 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.462979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.462396 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.462979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.462630 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.462979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.462649 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.462979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.462969 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.463459 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.463431 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.463562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.463507 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "kube-api-access-fg6hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.464285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.464260 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "kube-api-access-4vtf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.464552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.464527 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "kube-api-access-74ck8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.465343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.465306 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.466403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.466377 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.466685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.466661 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.466805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.466787 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.466849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.466811 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy" pod "1b6c8d4d-73ba-4bb3-9b73-251785be67d3" (UID: "1b6c8d4d-73ba-4bb3-9b73-251785be67d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.467562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.467533 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.467652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.467627 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm" pod "ae77f639-1cd8-4cb6-b543-736d30b20235" (UID: "ae77f639-1cd8-4cb6-b543-736d30b20235"). InnerVolumeSpecName "kube-api-access-dgzqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.467888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.467869 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.468526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.468510 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy" pod "ee5190d3-3e4b-4ed1-b72e-755df918cb31" (UID: "ee5190d3-3e4b-4ed1-b72e-755df918cb31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.468656 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.468643 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy" pod "a3ddf82f-c378-4f70-9352-a820f0dc0b12" (UID: "a3ddf82f-c378-4f70-9352-a820f0dc0b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.562529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562494 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562526 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562536 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6hd\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-kube-api-access-fg6hd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562548 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562558 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562567 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562575 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562586 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562595 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ck8\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-kube-api-access-74ck8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562604 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562613 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562621 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562629 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562637 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzqm\" (UniqueName: \"kubernetes.io/projected/ae77f639-1cd8-4cb6-b543-736d30b20235-kube-api-access-dgzqm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562646 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562653 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562661 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtf5\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-kube-api-access-4vtf5\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562670 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562678 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562686 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5190d3-3e4b-4ed1-b72e-755df918cb31-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562694 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562701 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5190d3-3e4b-4ed1-b72e-755df918cb31-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562711 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ddf82f-c378-4f70-9352-a820f0dc0b12-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562719 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562727 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562734 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae77f639-1cd8-4cb6-b543-736d30b20235-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562745 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae77f639-1cd8-4cb6-b543-736d30b20235-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.562783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.562754 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c8d4d-73ba-4bb3-9b73-251785be67d3-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.730092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.730019 5943 generic.go:356] "Generic (PLEG): container finished" podID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerID="3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183" exitCode=0 Oct 31 00:36:11.730363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.730338 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" Oct 31 00:36:11.730814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.730471 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" event={"ID":"ae77f639-1cd8-4cb6-b543-736d30b20235","Type":"ContainerDied","Data":"3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183"} Oct 31 00:36:11.730814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.730678 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq" event={"ID":"ae77f639-1cd8-4cb6-b543-736d30b20235","Type":"ContainerDied","Data":"70639acb6784ba016e17da01798a9ecc34f2376ade4e7c199cbdb67b630ea4fb"} Oct 31 00:36:11.730814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.730724 5943 scope.go:122] "RemoveContainer" containerID="3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183" Oct 31 00:36:11.736916 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.736883 5943 generic.go:356] "Generic (PLEG): container finished" podID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerID="386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991" exitCode=0 Oct 31 00:36:11.737024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.736985 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" event={"ID":"1b6c8d4d-73ba-4bb3-9b73-251785be67d3","Type":"ContainerDied","Data":"386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991"} Oct 31 00:36:11.737085 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.737033 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" event={"ID":"1b6c8d4d-73ba-4bb3-9b73-251785be67d3","Type":"ContainerDied","Data":"453f60fce6298e97f365272b8c9a432d4c72732919497c902d41a07fa09c51f5"} Oct 31 00:36:11.737152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.737123 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm" Oct 31 00:36:11.742751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.742712 5943 generic.go:356] "Generic (PLEG): container finished" podID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerID="d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa" exitCode=0 Oct 31 00:36:11.742877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.742807 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" Oct 31 00:36:11.742877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.742812 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" event={"ID":"a3ddf82f-c378-4f70-9352-a820f0dc0b12","Type":"ContainerDied","Data":"d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa"} Oct 31 00:36:11.742877 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.742874 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l" event={"ID":"a3ddf82f-c378-4f70-9352-a820f0dc0b12","Type":"ContainerDied","Data":"f47ff7164ec7e96e8e815c21b3a1ffc0ed448ce85bd08f36922e09de2b63b758"} Oct 31 00:36:11.747042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.747020 5943 generic.go:356] "Generic (PLEG): container finished" podID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerID="8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3" exitCode=0 Oct 31 00:36:11.747148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.747116 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" event={"ID":"ee5190d3-3e4b-4ed1-b72e-755df918cb31","Type":"ContainerDied","Data":"8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3"} Oct 31 00:36:11.747265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.747160 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" event={"ID":"ee5190d3-3e4b-4ed1-b72e-755df918cb31","Type":"ContainerDied","Data":"11dff64f6d316e2bc45a0793e7cd604bbe95ae51b01ad5748c6e870cdf1d282d"} Oct 31 00:36:11.747326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.747295 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t" Oct 31 00:36:11.759902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.759838 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:11.760832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.760791 5943 scope.go:122] "RemoveContainer" containerID="3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183" Oct 31 00:36:11.761169 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:11.761149 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183\": container with ID starting with 3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183 not found: ID does not exist" containerID="3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183" Oct 31 00:36:11.761169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.761225 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183"} err="failed to get container status \"3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183\": rpc error: code = NotFound desc = could not find container \"3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183\": container with ID starting with 3d9523ad5b19318c84bc0f657d206eca8b0408e0575d8a950217de8382828183 not found: ID does not exist" Oct 31 00:36:11.761451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.761250 5943 scope.go:122] "RemoveContainer" containerID="386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991" Oct 31 00:36:11.778342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.778314 5943 scope.go:122] "RemoveContainer" containerID="386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991" Oct 31 00:36:11.778772 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:11.778747 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991\": container with ID starting with 386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991 not found: ID does not exist" containerID="386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991" Oct 31 00:36:11.778896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.778776 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991"} err="failed to get container status \"386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991\": rpc error: code = NotFound desc = could not find container \"386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991\": container with ID starting with 386da22ff73f79fb545d4464a9cc7e7f60748763fdc660e47f94efff8245e991 not found: ID does not exist" Oct 31 00:36:11.778896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.778804 5943 scope.go:122] "RemoveContainer" containerID="d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa" Oct 31 00:36:11.791920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.791520 5943 scope.go:122] "RemoveContainer" containerID="d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa" Oct 31 00:36:11.793067 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:11.792052 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa\": container with ID starting with d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa not found: ID does not exist" containerID="d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa" Oct 31 00:36:11.793207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.793128 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa"} err="failed to get container status \"d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa\": rpc error: code = NotFound desc = could not find container \"d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa\": container with ID starting with d5d0638d36ff21148d179759df1e0cad00214bf8c3552403d874367d4e8701fa not found: ID does not exist" Oct 31 00:36:11.793273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.793169 5943 scope.go:122] "RemoveContainer" containerID="8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3" Oct 31 00:36:11.803527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.803509 5943 scope.go:122] "RemoveContainer" containerID="8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3" Oct 31 00:36:11.803871 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:11.803846 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3\": container with ID starting with 8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3 not found: ID does not exist" containerID="8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3" Oct 31 00:36:11.803959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.803872 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3"} err="failed to get container status \"8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3\": rpc error: code = NotFound desc = could not find container \"8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3\": container with ID starting with 8bc6156b6fa6434e4856c76aa78397a2bf405791a3a885457ba91ff1126cc4d3 not found: ID does not exist" Oct 31 00:36:11.929501 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.929420 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l"] Oct 31 00:36:11.936545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:11.936516 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-9jn9l"] Oct 31 00:36:12.044199 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.044150 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" path="/var/lib/kubelet/pods/3218053c-1eb0-4bc8-9c6e-9654627540b2/volumes" Oct 31 00:36:12.044956 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.044930 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" path="/var/lib/kubelet/pods/a3ddf82f-c378-4f70-9352-a820f0dc0b12/volumes" Oct 31 00:36:12.046265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.046245 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq"] Oct 31 00:36:12.053027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.052998 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-knvnq"] Oct 31 00:36:12.157839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.157807 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds"] Oct 31 00:36:12.158098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.158073 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" containerID="cri-o://aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4" gracePeriod=30 Oct 31 00:36:12.181717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.179972 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t"] Oct 31 00:36:12.191911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.191881 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-q2b2t"] Oct 31 00:36:12.310281 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.306980 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm"] Oct 31 00:36:12.316590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.312748 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gltnm"] Oct 31 00:36:12.358742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.358714 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:36:12.459124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.459097 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:36:12.493021 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.492983 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493021 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493024 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493104 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493141 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493211 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493256 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493493 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493303 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493493 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493380 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.493493 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493398 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493493 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493438 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket\") pod \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\" (UID: \"8aa478f7-2f67-48ff-b604-b0716fe8e55e\") " Oct 31 00:36:12.493669 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493501 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.493769 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.493742 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.494229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.494115 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.494229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.494126 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.494229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.494149 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.494229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.494163 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.494441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.494229 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.495594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.495573 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:12.495668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.495599 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.496108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.496088 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.496161 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.496110 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq" pod "8aa478f7-2f67-48ff-b604-b0716fe8e55e" (UID: "8aa478f7-2f67-48ff-b604-b0716fe8e55e"). InnerVolumeSpecName "kube-api-access-jjvbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.541379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.541339 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:12.595443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595400 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595461 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595501 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595545 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595626 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595675 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595712 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595836 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.595994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.595856 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy\") pod \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\" (UID: \"4c9169c6-9490-4291-9369-1fd3ac5dafbe\") " Oct 31 00:36:12.596318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.596293 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.596440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.596322 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.596440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.596336 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.596440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.596349 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.596440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.596362 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa478f7-2f67-48ff-b604-b0716fe8e55e-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.596440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.596374 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvbq\" (UniqueName: \"kubernetes.io/projected/8aa478f7-2f67-48ff-b604-b0716fe8e55e-kube-api-access-jjvbq\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.597140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.597104 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.597376 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.597353 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.597948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.597802 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.597948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.597901 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.597948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.597927 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.599345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.599319 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.599598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.599578 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:12.599666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.599608 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.599950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.599930 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74" pod "4c9169c6-9490-4291-9369-1fd3ac5dafbe" (UID: "4c9169c6-9490-4291-9369-1fd3ac5dafbe"). InnerVolumeSpecName "kube-api-access-4hw74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.697855 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697811 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.697855 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697848 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.697855 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697861 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.698090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697874 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.698090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697896 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.698090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697908 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.698090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697919 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9169c6-9490-4291-9369-1fd3ac5dafbe-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.698090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697935 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hw74\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-kube-api-access-4hw74\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.698090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.697948 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9169c6-9490-4291-9369-1fd3ac5dafbe-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.759678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.759639 5943 generic.go:356] "Generic (PLEG): container finished" podID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerID="8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa" exitCode=0 Oct 31 00:36:12.759886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.759694 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" Oct 31 00:36:12.759886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.759731 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" event={"ID":"4c9169c6-9490-4291-9369-1fd3ac5dafbe","Type":"ContainerDied","Data":"8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa"} Oct 31 00:36:12.759886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.759835 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2" event={"ID":"4c9169c6-9490-4291-9369-1fd3ac5dafbe","Type":"ContainerDied","Data":"7e61bb314c2f946152c39a16d177c15942033a1cef7568e4502f90deef5b530d"} Oct 31 00:36:12.759886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.759870 5943 scope.go:122] "RemoveContainer" containerID="8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa" Oct 31 00:36:12.761829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.761779 5943 generic.go:356] "Generic (PLEG): container finished" podID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerID="8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a" exitCode=0 Oct 31 00:36:12.762039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.761900 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" event={"ID":"8aa478f7-2f67-48ff-b604-b0716fe8e55e","Type":"ContainerDied","Data":"8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a"} Oct 31 00:36:12.762039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.761944 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" event={"ID":"8aa478f7-2f67-48ff-b604-b0716fe8e55e","Type":"ContainerDied","Data":"5a80aee0f7bcd870ed787372cb5654607aa32068cc3f2df00c4253b9156831ac"} Oct 31 00:36:12.762039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.762032 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw" Oct 31 00:36:12.774099 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.774077 5943 scope.go:122] "RemoveContainer" containerID="8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa" Oct 31 00:36:12.774509 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:12.774452 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa\": container with ID starting with 8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa not found: ID does not exist" containerID="8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa" Oct 31 00:36:12.774509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.774488 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa"} err="failed to get container status \"8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa\": rpc error: code = NotFound desc = could not find container \"8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa\": container with ID starting with 8a0072ff397734a8ed91a489f7d50272b1057a48b86ee5d2a4427842691349aa not found: ID does not exist" Oct 31 00:36:12.774509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.774510 5943 scope.go:122] "RemoveContainer" containerID="8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a" Oct 31 00:36:12.785018 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.784990 5943 scope.go:122] "RemoveContainer" containerID="8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a" Oct 31 00:36:12.785579 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:12.785543 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a\": container with ID starting with 8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a not found: ID does not exist" containerID="8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a" Oct 31 00:36:12.785579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.785593 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a"} err="failed to get container status \"8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a\": rpc error: code = NotFound desc = could not find container \"8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a\": container with ID starting with 8b746ba44597d58095b19bfe6fa94ee720a7417be3613abbe2fa69174746425a not found: ID does not exist" Oct 31 00:36:12.856938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.856905 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:12.956744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.956706 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw"] Oct 31 00:36:12.963153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:12.963122 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rqmlw"] Oct 31 00:36:13.030083 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:13.029987 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:13.072422 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:13.072387 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2"] Oct 31 00:36:13.083814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:13.083779 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-z5fl2"] Oct 31 00:36:14.042209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:14.042148 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" path="/var/lib/kubelet/pods/1b6c8d4d-73ba-4bb3-9b73-251785be67d3/volumes" Oct 31 00:36:14.042681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:14.042666 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" path="/var/lib/kubelet/pods/4c9169c6-9490-4291-9369-1fd3ac5dafbe/volumes" Oct 31 00:36:14.043103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:14.043090 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" path="/var/lib/kubelet/pods/8aa478f7-2f67-48ff-b604-b0716fe8e55e/volumes" Oct 31 00:36:14.043842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:14.043827 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" path="/var/lib/kubelet/pods/ae77f639-1cd8-4cb6-b543-736d30b20235/volumes" Oct 31 00:36:14.044269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:14.044256 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" path="/var/lib/kubelet/pods/ee5190d3-3e4b-4ed1-b72e-755df918cb31/volumes" Oct 31 00:36:15.901824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:15.901781 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:16.099083 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:16.099050 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:16.175033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:16.174937 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:16.175033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:16.175014 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:16.515368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:16.508477 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:17.489794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.489763 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:36:17.579556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579519 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.579758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579607 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.579758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579747 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.579863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579777 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.579863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579855 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.580002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579926 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.580002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579930 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:17.580002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.579957 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.580158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580002 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.580158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580044 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:17.580158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580067 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert\") pod \"9624f662-6a04-4586-b941-ba89204173ba\" (UID: \"9624f662-6a04-4586-b941-ba89204173ba\") " Oct 31 00:36:17.580158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580135 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:17.580537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580307 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:17.580537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580498 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:17.580854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580830 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9624f662-6a04-4586-b941-ba89204173ba-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.580923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580860 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.580923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580877 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.580923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580893 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.580923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.580908 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.582348 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.582323 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "kube-api-access-x2vdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:17.582854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.582827 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:17.583098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.583073 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:17.583329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.583309 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy" pod "9624f662-6a04-4586-b941-ba89204173ba" (UID: "9624f662-6a04-4586-b941-ba89204173ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:17.682590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.682548 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9624f662-6a04-4586-b941-ba89204173ba-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.682590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.682597 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2vdc\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-kube-api-access-x2vdc\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.682590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.682616 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9624f662-6a04-4586-b941-ba89204173ba-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.682927 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.682630 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9624f662-6a04-4586-b941-ba89204173ba-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:17.706460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.706272 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:17.800221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.800153 5943 generic.go:356] "Generic (PLEG): container finished" podID="9624f662-6a04-4586-b941-ba89204173ba" containerID="aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4" exitCode=0 Oct 31 00:36:17.800437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.800245 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" Oct 31 00:36:17.800437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.800243 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" event={"ID":"9624f662-6a04-4586-b941-ba89204173ba","Type":"ContainerDied","Data":"aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4"} Oct 31 00:36:17.800437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.800312 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds" event={"ID":"9624f662-6a04-4586-b941-ba89204173ba","Type":"ContainerDied","Data":"e2bc9da3b261ed997f4287bce4c19c5f00b07f9a2f9b21e4d40161d597d9145c"} Oct 31 00:36:17.800437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.800334 5943 scope.go:122] "RemoveContainer" containerID="aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4" Oct 31 00:36:17.825401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.824677 5943 scope.go:122] "RemoveContainer" containerID="aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4" Oct 31 00:36:17.825401 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:17.825093 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4\": container with ID starting with aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4 not found: ID does not exist" containerID="aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4" Oct 31 00:36:17.825401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.825125 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4"} err="failed to get container status \"aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4\": rpc error: code = NotFound desc = could not find container \"aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4\": container with ID starting with aba615d86d3dc2df27ddaea73b50087e909914e18d32b1c56fc47f75649df4e4 not found: ID does not exist" Oct 31 00:36:17.967153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.967119 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds"] Oct 31 00:36:17.971048 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:17.971016 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-5nbds"] Oct 31 00:36:18.045106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:18.045072 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9624f662-6a04-4586-b941-ba89204173ba" path="/var/lib/kubelet/pods/9624f662-6a04-4586-b941-ba89204173ba/volumes" Oct 31 00:36:18.073278 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:18.073145 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:18.362146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:18.362053 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:18.592601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:18.592567 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5"] Oct 31 00:36:18.595104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:18.595074 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7872-hhft5"] Oct 31 00:36:19.046412 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:19.046375 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:19.240289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:19.240255 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:19.449465 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:19.449382 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:19.596962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:19.596887 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:19.778724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:19.778687 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds"] Oct 31 00:36:19.827401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:19.827371 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a59a7-pc8ds"] Oct 31 00:36:20.051020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:20.045425 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1621cbb8-1ba0-4b8f-9740-cff0f15f11a2" path="/var/lib/kubelet/pods/1621cbb8-1ba0-4b8f-9740-cff0f15f11a2/volumes" Oct 31 00:36:20.051020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:20.046163 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dec3283-9ed0-4b54-84cf-fbe375295292" path="/var/lib/kubelet/pods/3dec3283-9ed0-4b54-84cf-fbe375295292/volumes" Oct 31 00:36:20.229701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:20.229659 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:24.758734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:24.758700 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:24.978742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:24.978703 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:25.131056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:25.130958 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:25.390908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:25.390811 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:26.270641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.270610 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg"] Oct 31 00:36:26.271141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271128 5943 cpu_manager.go:414] "containerMap: removing container" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271142 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271171 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271190 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271196 5943 cpu_manager.go:414] "containerMap: removing container" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271201 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271208 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" Oct 31 00:36:26.271250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271213 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271631 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271712 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271755 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271763 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271780 5943 cpu_manager.go:414] "containerMap: removing container" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271790 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271836 5943 cpu_manager.go:414] "containerMap: removing container" podUID="3dec3283-9ed0-4b54-84cf-fbe375295292" containerName="revoke-keys" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271845 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dec3283-9ed0-4b54-84cf-fbe375295292" containerName="revoke-keys" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271866 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271873 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271893 5943 cpu_manager.go:414] "containerMap: removing container" podUID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" containerName="main" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.271900 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" containerName="main" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272317 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dec3283-9ed0-4b54-84cf-fbe375295292" containerName="revoke-keys" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272346 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9624f662-6a04-4586-b941-ba89204173ba" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272356 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="a131f50e-c5e7-4f3f-a0d7-a89df8dd973e" containerName="main" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272377 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9169c6-9490-4291-9369-1fd3ac5dafbe" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272395 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa478f7-2f67-48ff-b604-b0716fe8e55e" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272419 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ddf82f-c378-4f70-9352-a820f0dc0b12" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272428 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5190d3-3e4b-4ed1-b72e-755df918cb31" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272447 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6c8d4d-73ba-4bb3-9b73-251785be67d3" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272463 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae77f639-1cd8-4cb6-b543-736d30b20235" containerName="istio-proxy" Oct 31 00:36:26.274959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.272473 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="3218053c-1eb0-4bc8-9c6e-9654627540b2" containerName="istio-proxy" Oct 31 00:36:26.279449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.278754 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.310159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.310129 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.310327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.310227 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.310372 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.310326 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp26l\" (UniqueName: \"kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.333618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.333586 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg"] Oct 31 00:36:26.379562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.379505 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:26.379752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.379733 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:36:26.379825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.379806 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:36:26.380596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.380581 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:36:26.411837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.411799 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp26l\" (UniqueName: \"kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.411990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.411887 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.411990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.411968 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.412556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.412533 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.414593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.414571 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.474719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.474683 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp26l\" (UniqueName: \"kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l\") pod \"maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.594584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.594503 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:26.763638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.763601 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:26.923363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.923268 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:26.979285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:26.979263 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg"] Oct 31 00:36:26.981401 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:36:26.981372 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd555ebb0_b554_43b1_962c_e70e53da63c7.slice/crio-70c8fdf7b032f3ba8653f11d1070165aea9473bfb8688108f9ae758daada0de5 WatchSource:0}: Error finding container 70c8fdf7b032f3ba8653f11d1070165aea9473bfb8688108f9ae758daada0de5: Status 404 returned error can't find the container with id 70c8fdf7b032f3ba8653f11d1070165aea9473bfb8688108f9ae758daada0de5 Oct 31 00:36:27.165014 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:27.164985 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:27.480739 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:27.477991 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:36:27.869030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:27.868991 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" event={"ID":"d555ebb0-b554-43b1-962c-e70e53da63c7","Type":"ContainerStarted","Data":"54fdae4924d5cc4ba35ecf60a24974c26c5196ff71966e2865a839fdd45e56f1"} Oct 31 00:36:27.869242 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:27.869053 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" event={"ID":"d555ebb0-b554-43b1-962c-e70e53da63c7","Type":"ContainerStarted","Data":"70c8fdf7b032f3ba8653f11d1070165aea9473bfb8688108f9ae758daada0de5"} Oct 31 00:36:27.984039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:27.983983 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" podStartSLOduration=1.983969123 podStartE2EDuration="1.983969123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:27.983158748 +0000 UTC m=+1968.076448960" watchObservedRunningTime="2026-10-31 00:36:27.983969123 +0000 UTC m=+1968.077259309" Oct 31 00:36:28.299080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:28.299046 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:28.876760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:28.876710 5943 generic.go:356] "Generic (PLEG): container finished" podID="d555ebb0-b554-43b1-962c-e70e53da63c7" containerID="54fdae4924d5cc4ba35ecf60a24974c26c5196ff71966e2865a839fdd45e56f1" exitCode=0 Oct 31 00:36:28.877258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:28.876850 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" event={"ID":"d555ebb0-b554-43b1-962c-e70e53da63c7","Type":"ContainerDied","Data":"54fdae4924d5cc4ba35ecf60a24974c26c5196ff71966e2865a839fdd45e56f1"} Oct 31 00:36:28.979441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:28.979400 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:29.584361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:29.584318 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:30.077619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.077594 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:30.081844 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.081813 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:30.163587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.163496 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca\") pod \"d555ebb0-b554-43b1-962c-e70e53da63c7\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " Oct 31 00:36:30.163746 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.163653 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca\") pod \"d555ebb0-b554-43b1-962c-e70e53da63c7\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " Oct 31 00:36:30.163746 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.163688 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l\" (UniqueName: \"kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l\") pod \"d555ebb0-b554-43b1-962c-e70e53da63c7\" (UID: \"d555ebb0-b554-43b1-962c-e70e53da63c7\") " Oct 31 00:36:30.163965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.163935 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca" pod "d555ebb0-b554-43b1-962c-e70e53da63c7" (UID: "d555ebb0-b554-43b1-962c-e70e53da63c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:30.164283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.164261 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.166418 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.166388 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l" pod "d555ebb0-b554-43b1-962c-e70e53da63c7" (UID: "d555ebb0-b554-43b1-962c-e70e53da63c7"). InnerVolumeSpecName "kube-api-access-dp26l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:30.168370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.168342 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca" pod "d555ebb0-b554-43b1-962c-e70e53da63c7" (UID: "d555ebb0-b554-43b1-962c-e70e53da63c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:30.265676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.265621 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d555ebb0-b554-43b1-962c-e70e53da63c7-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.265676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.265670 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp26l\" (UniqueName: \"kubernetes.io/projected/d555ebb0-b554-43b1-962c-e70e53da63c7-kube-api-access-dp26l\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.892069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.892031 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" Oct 31 00:36:30.892069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.892059 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg" event={"ID":"d555ebb0-b554-43b1-962c-e70e53da63c7","Type":"ContainerDied","Data":"70c8fdf7b032f3ba8653f11d1070165aea9473bfb8688108f9ae758daada0de5"} Oct 31 00:36:30.892360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:30.892106 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c8fdf7b032f3ba8653f11d1070165aea9473bfb8688108f9ae758daada0de5" Oct 31 00:36:32.807772 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:32.807735 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.104299 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:33.104216 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.297630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:33.297597 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:35.129940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:35.129905 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:35.316924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:35.316885 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:35.462473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:35.462373 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:38.034948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:38.034910 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:38.237204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:38.233808 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:39.241241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:39.241203 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:36:40.553028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:40.552999 5943 scope.go:122] "RemoveContainer" containerID="605b7095697c40039fc8a492f44f633cee3f38527a65ae3e6d386905754d8b7f" Oct 31 00:36:40.723072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:40.723035 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:41.539511 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:41.539476 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:41.711369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:41.711333 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:47.462800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:47.462760 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:49.344945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:49.344855 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:52.203105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:52.203064 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:57.164104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:57.164066 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:57.484549 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:36:57.484450 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:36:59.994135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:36:59.994092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:03.641275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:03.641231 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:08.408156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:08.408117 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:08.558671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:08.558636 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:08.704896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:08.704797 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:13.433904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:13.433856 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:17.050394 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:17.050350 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:18.294039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:18.294000 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:21.891063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:21.891024 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:37:23.072261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:23.072222 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.477468 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:37:27.477427 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:37:27.732757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:27.732659 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.926100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:27.926058 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:28.127582 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:28.127539 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:28.853448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:28.853410 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:32.724029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:32.723987 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:37.634951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:37.634911 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.234723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:39.234672 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.453952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:39.453923 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.641287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:39.641148 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.801231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:39.800437 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.955574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:39.955471 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:44.454254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:44.454214 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:49.309264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:49.309226 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:50.279448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:50.279409 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:37:50.658748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:50.658632 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.379828 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:37:54.379783 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:57.477249 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:37:57.477205 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:38:00.867801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:00.867764 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:05.681464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:05.681427 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:10.530509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:10.530470 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:15.466770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:15.466728 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:15.748244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:15.748132 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:16.301831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:16.301793 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.004624 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:17.004594 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.535499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:17.535458 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.240931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:20.240732 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:21.071505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:21.071465 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw"] Oct 31 00:38:21.071733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:21.071710 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" containerID="cri-o://cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc" gracePeriod=30 Oct 31 00:38:21.917031 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:21.916994 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.209940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:22.209829 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.355942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:22.355903 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.429225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:22.429173 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg"] Oct 31 00:38:22.478304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:22.478207 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cda64-rvxkg"] Oct 31 00:38:22.825607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:22.825568 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.984891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:22.984853 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:23.145914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:23.145803 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:24.042874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:24.042839 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d555ebb0-b554-43b1-962c-e70e53da63c7" path="/var/lib/kubelet/pods/d555ebb0-b554-43b1-962c-e70e53da63c7/volumes" Oct 31 00:38:25.866104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:25.866068 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:38:25.866544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:25.866130 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:38:26.094361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.094335 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:26.270795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.270751 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:26.412275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.412250 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:38:26.480115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480083 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480133 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480156 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480242 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480350 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480421 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480476 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480536 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:26.480668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480586 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:26.480668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480596 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480729 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy\") pod \"9999ba88-9551-4b1d-b54b-87b509350556\" (UID: \"9999ba88-9551-4b1d-b54b-87b509350556\") " Oct 31 00:38:26.480822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480788 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:26.480822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480785 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:26.480822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.480802 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:26.482134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.482023 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.482134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.482059 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.482134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.482074 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.482134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.482087 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.482134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.482107 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9999ba88-9551-4b1d-b54b-87b509350556-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.484468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.483388 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:26.484595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.484541 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:26.493244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.490284 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:38:26.494851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.494820 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl" pod "9999ba88-9551-4b1d-b54b-87b509350556" (UID: "9999ba88-9551-4b1d-b54b-87b509350556"). InnerVolumeSpecName "kube-api-access-45sbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:26.583451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.583417 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9999ba88-9551-4b1d-b54b-87b509350556-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.583451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.583448 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9999ba88-9551-4b1d-b54b-87b509350556-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.583659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.583458 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45sbl\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-kube-api-access-45sbl\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.583659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.583468 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9999ba88-9551-4b1d-b54b-87b509350556-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:26.635096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.635010 5943 generic.go:356] "Generic (PLEG): container finished" podID="9999ba88-9551-4b1d-b54b-87b509350556" containerID="cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc" exitCode=0 Oct 31 00:38:26.635096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.635071 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" Oct 31 00:38:26.635096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.635088 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" event={"ID":"9999ba88-9551-4b1d-b54b-87b509350556","Type":"ContainerDied","Data":"cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc"} Oct 31 00:38:26.635361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.635172 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw" event={"ID":"9999ba88-9551-4b1d-b54b-87b509350556","Type":"ContainerDied","Data":"33e20c24497bc60dc98a10e82ae92d71f81c54c55a92cc9ccb4f0a3effefe241"} Oct 31 00:38:26.635361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.635218 5943 scope.go:122] "RemoveContainer" containerID="cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc" Oct 31 00:38:26.648586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.648233 5943 scope.go:122] "RemoveContainer" containerID="cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc" Oct 31 00:38:26.649577 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:38:26.649539 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc\": container with ID starting with cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc not found: ID does not exist" containerID="cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc" Oct 31 00:38:26.649685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.649590 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc"} err="failed to get container status \"cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc\": rpc error: code = NotFound desc = could not find container \"cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc\": container with ID starting with cb403d27734f82dc59779c76a12ab470f9003f29decfdde9f4990603859907dc not found: ID does not exist" Oct 31 00:38:26.807058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.807022 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw"] Oct 31 00:38:26.812107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:26.812074 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-sddxw"] Oct 31 00:38:27.456594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:27.456551 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:27.479159 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:38:27.479112 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:38:27.947618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:27.947585 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:38:28.043254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:28.043220 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9999ba88-9551-4b1d-b54b-87b509350556" path="/var/lib/kubelet/pods/9999ba88-9551-4b1d-b54b-87b509350556/volumes" Oct 31 00:38:29.739793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:29.739759 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:29.962973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:29.962933 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:30.111769 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:30.111722 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd"] Oct 31 00:38:30.112046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:30.112008 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" containerID="cri-o://8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3" gracePeriod=30 Oct 31 00:38:30.130736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:30.130697 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:32.676650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:32.676610 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:34.098480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.098443 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz"] Oct 31 00:38:34.099027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.099012 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" Oct 31 00:38:34.099073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.099028 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" Oct 31 00:38:34.099073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.099042 5943 cpu_manager.go:414] "containerMap: removing container" podUID="d555ebb0-b554-43b1-962c-e70e53da63c7" containerName="revoke-keys" Oct 31 00:38:34.099073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.099047 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d555ebb0-b554-43b1-962c-e70e53da63c7" containerName="revoke-keys" Oct 31 00:38:34.099206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.099116 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9999ba88-9551-4b1d-b54b-87b509350556" containerName="istio-proxy" Oct 31 00:38:34.099206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.099127 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="d555ebb0-b554-43b1-962c-e70e53da63c7" containerName="revoke-keys" Oct 31 00:38:34.102262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.102245 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.166561 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.166521 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.166752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.166583 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77dmx\" (UniqueName: \"kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.166890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.166872 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.204341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.204307 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:38:34.204651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.204399 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:38:34.204651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.204452 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:38:34.207215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.207167 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz"] Oct 31 00:38:34.254735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.254701 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:38:34.267661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.267628 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.267866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.267691 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.267866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.267720 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77dmx\" (UniqueName: \"kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.268410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.268383 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.270415 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.270394 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.329247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.329212 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77dmx\" (UniqueName: \"kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx\") pod \"maas-api-revoke-sub-dbc33e50-w4zqz\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.416831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.416747 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:34.805532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:34.805502 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz"] Oct 31 00:38:34.807873 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:38:34.807843 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a400d_a0ff_4e7c_8e0c_88c68ff8ac64.slice/crio-5b23b8f569fb1c2f099de25f2c7a95d67a22d3ddba147ea69b1cd01cb01063dd WatchSource:0}: Error finding container 5b23b8f569fb1c2f099de25f2c7a95d67a22d3ddba147ea69b1cd01cb01063dd: Status 404 returned error can't find the container with id 5b23b8f569fb1c2f099de25f2c7a95d67a22d3ddba147ea69b1cd01cb01063dd Oct 31 00:38:35.387938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.387870 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:38:35.483841 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.483811 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.483870 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.483916 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.483941 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.483999 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484079 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484153 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484167 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:35.484337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484217 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:35.484337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484227 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.484337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484262 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data\") pod \"abd292ef-804f-4898-8e7b-0db768cd0930\" (UID: \"abd292ef-804f-4898-8e7b-0db768cd0930\") " Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484517 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484558 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484746 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.484989 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.485080 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd292ef-804f-4898-8e7b-0db768cd0930-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.485097 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.485111 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.485052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.485126 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.486627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.486604 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "kube-api-access-tdv7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:35.486857 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.486841 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:35.487140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.487115 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:38:35.487262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.487120 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token" pod "abd292ef-804f-4898-8e7b-0db768cd0930" (UID: "abd292ef-804f-4898-8e7b-0db768cd0930"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:35.586562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.586516 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd292ef-804f-4898-8e7b-0db768cd0930-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.586562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.586560 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdv7j\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-kube-api-access-tdv7j\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.586768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.586579 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd292ef-804f-4898-8e7b-0db768cd0930-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.586768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.586594 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd292ef-804f-4898-8e7b-0db768cd0930-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:35.771803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.771756 5943 generic.go:356] "Generic (PLEG): container finished" podID="2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" containerID="f6823099e8612c19d69c6c8587be0d928c984f963a3443c3102b2ef0386a0368" exitCode=0 Oct 31 00:38:35.771981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.771840 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" event={"ID":"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64","Type":"ContainerDied","Data":"f6823099e8612c19d69c6c8587be0d928c984f963a3443c3102b2ef0386a0368"} Oct 31 00:38:35.771981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.771930 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" event={"ID":"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64","Type":"ContainerStarted","Data":"5b23b8f569fb1c2f099de25f2c7a95d67a22d3ddba147ea69b1cd01cb01063dd"} Oct 31 00:38:35.773699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.773675 5943 generic.go:356] "Generic (PLEG): container finished" podID="abd292ef-804f-4898-8e7b-0db768cd0930" containerID="8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3" exitCode=0 Oct 31 00:38:35.773849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.773710 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" event={"ID":"abd292ef-804f-4898-8e7b-0db768cd0930","Type":"ContainerDied","Data":"8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3"} Oct 31 00:38:35.773849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.773728 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" Oct 31 00:38:35.773849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.773748 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd" event={"ID":"abd292ef-804f-4898-8e7b-0db768cd0930","Type":"ContainerDied","Data":"705bf462a5864f5072b58bddf75b9d265571de8121d49b56f56680f91fa10e27"} Oct 31 00:38:35.773849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.773771 5943 scope.go:122] "RemoveContainer" containerID="8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3" Oct 31 00:38:35.784520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.784502 5943 scope.go:122] "RemoveContainer" containerID="046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6" Oct 31 00:38:35.794073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.794049 5943 scope.go:122] "RemoveContainer" containerID="8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3" Oct 31 00:38:35.794469 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:38:35.794444 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3\": container with ID starting with 8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3 not found: ID does not exist" containerID="8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3" Oct 31 00:38:35.794564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.794483 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3"} err="failed to get container status \"8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3\": rpc error: code = NotFound desc = could not find container \"8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3\": container with ID starting with 8dd1f68317e31337d081e9a13ddbe4ecc9f97b509b00e18677ce94bfd3f5e8f3 not found: ID does not exist" Oct 31 00:38:35.794564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.794509 5943 scope.go:122] "RemoveContainer" containerID="046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6" Oct 31 00:38:35.794772 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:38:35.794754 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6\": container with ID starting with 046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6 not found: ID does not exist" containerID="046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6" Oct 31 00:38:35.794815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.794780 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6"} err="failed to get container status \"046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6\": rpc error: code = NotFound desc = could not find container \"046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6\": container with ID starting with 046d88a345ebbe63e557225e85b997cf98b93a64f2980f027ba25effbbd23be6 not found: ID does not exist" Oct 31 00:38:35.998806 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:35.998719 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd"] Oct 31 00:38:36.005382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:36.005351 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2cda64-openshift-default-dbd549c6c-nvwzd"] Oct 31 00:38:36.042301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:36.042271 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" path="/var/lib/kubelet/pods/abd292ef-804f-4898-8e7b-0db768cd0930/volumes" Oct 31 00:38:36.050496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:36.050466 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:36.533735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:36.533649 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:36.946996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:36.946965 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:37.004087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.004053 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca\") pod \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " Oct 31 00:38:37.004313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.004244 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx\" (UniqueName: \"kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx\") pod \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " Oct 31 00:38:37.004371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.004315 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca\") pod \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\" (UID: \"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64\") " Oct 31 00:38:37.004706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.004673 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca" pod "2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" (UID: "2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:37.006845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.006818 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca" pod "2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" (UID: "2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:37.006845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.006839 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx" pod "2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" (UID: "2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64"). InnerVolumeSpecName "kube-api-access-77dmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:37.024731 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.024687 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:37.106329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.106246 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:37.106329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.106278 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77dmx\" (UniqueName: \"kubernetes.io/projected/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-kube-api-access-77dmx\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:37.106329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.106288 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:37.115227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.115192 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp"] Oct 31 00:38:37.115430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.115404 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" containerID="cri-o://0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9" gracePeriod=30 Oct 31 00:38:37.225610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.225569 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:37.783817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.783784 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:37.790054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.790030 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" Oct 31 00:38:37.790054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.790040 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz" event={"ID":"2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64","Type":"ContainerDied","Data":"5b23b8f569fb1c2f099de25f2c7a95d67a22d3ddba147ea69b1cd01cb01063dd"} Oct 31 00:38:37.790277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.790100 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b23b8f569fb1c2f099de25f2c7a95d67a22d3ddba147ea69b1cd01cb01063dd" Oct 31 00:38:37.988461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:37.988427 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:38.841889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:38.841845 5943 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qq4mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:38:38.842322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:38.841916 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:38:39.865270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:39.865227 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:42.239702 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.239644 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:42.393352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.393290 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:38:42.444949 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.444850 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502098 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502166 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502291 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502325 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502395 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502479 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502545 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502572 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.502688 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket\") pod \"50c327e5-d646-4e68-9c84-0c94c947d5d3\" (UID: \"50c327e5-d646-4e68-9c84-0c94c947d5d3\") " Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.503843 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.504276 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:42.505202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.504553 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:42.505957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.505347 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:42.505957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.505517 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:42.508835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.508802 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "kube-api-access-mtrc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:42.509269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.509247 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:42.509556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.509538 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:42.510266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.510239 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo" pod "50c327e5-d646-4e68-9c84-0c94c947d5d3" (UID: "50c327e5-d646-4e68-9c84-0c94c947d5d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:38:42.604425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604393 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c327e5-d646-4e68-9c84-0c94c947d5d3-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604421 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604430 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604439 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604448 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604456 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604465 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604473 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtrc7\" (UniqueName: \"kubernetes.io/projected/50c327e5-d646-4e68-9c84-0c94c947d5d3-kube-api-access-mtrc7\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.604662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.604483 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c327e5-d646-4e68-9c84-0c94c947d5d3-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.607330 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.607288 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:42.823615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.823578 5943 generic.go:356] "Generic (PLEG): container finished" podID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerID="0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9" exitCode=0 Oct 31 00:38:42.823803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.823629 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" Oct 31 00:38:42.823803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.823666 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" event={"ID":"50c327e5-d646-4e68-9c84-0c94c947d5d3","Type":"ContainerDied","Data":"0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9"} Oct 31 00:38:42.823803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.823728 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp" event={"ID":"50c327e5-d646-4e68-9c84-0c94c947d5d3","Type":"ContainerDied","Data":"577869e274a43bd859d5f6c666b85d6dcabbf536ce6981723d099c84a276d2f7"} Oct 31 00:38:42.823803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.823757 5943 scope.go:122] "RemoveContainer" containerID="0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9" Oct 31 00:38:42.834860 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.834832 5943 scope.go:122] "RemoveContainer" containerID="0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9" Oct 31 00:38:42.835224 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:38:42.835201 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9\": container with ID starting with 0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9 not found: ID does not exist" containerID="0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9" Oct 31 00:38:42.835269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.835236 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9"} err="failed to get container status \"0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9\": rpc error: code = NotFound desc = could not find container \"0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9\": container with ID starting with 0533e688f5ad5390740647c9260441730f4499d71a19fef72e104f1469b04aa9 not found: ID does not exist" Oct 31 00:38:42.990703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.990667 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp"] Oct 31 00:38:42.996403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:42.996375 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qq4mp"] Oct 31 00:38:44.040964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:44.040931 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" path="/var/lib/kubelet/pods/50c327e5-d646-4e68-9c84-0c94c947d5d3/volumes" Oct 31 00:38:45.460456 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:45.460420 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.674650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:45.674616 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:46.870248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:46.870205 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:47.019222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:47.019165 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:47.821887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:47.821846 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:47.978018 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:47.977977 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:50.050784 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:50.050749 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:51.892811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:51.892772 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:52.224466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:52.224383 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:52.393896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:52.393846 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:52.541112 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:52.541075 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:52.955599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:52.955505 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:54.713684 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:54.713649 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:54.964365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:54.964276 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:55.484440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:55.484403 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:55.674257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:55.674219 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:56.902127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:56.902086 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:57.292889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:57.292856 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:57.477949 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:38:57.477906 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:38:59.009684 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:59.009642 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:59.253078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:59.253047 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:59.415879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:38:59.415782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:02.320592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:02.320559 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:02.496077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:02.496040 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.312550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:04.312510 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:09.351995 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.351957 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:09.905489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.905450 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8"] Oct 31 00:39:09.906023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906008 5943 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" containerName="revoke-keys" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906025 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" containerName="revoke-keys" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906038 5943 cpu_manager.go:414] "containerMap: removing container" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906043 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906051 5943 cpu_manager.go:414] "containerMap: removing container" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906057 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906064 5943 cpu_manager.go:414] "containerMap: removing container" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" Oct 31 00:39:09.906074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906069 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" Oct 31 00:39:09.906323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906144 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" containerName="revoke-keys" Oct 31 00:39:09.906323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906154 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c327e5-d646-4e68-9c84-0c94c947d5d3" containerName="istio-proxy" Oct 31 00:39:09.906323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906160 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" Oct 31 00:39:09.906323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.906167 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd292ef-804f-4898-8e7b-0db768cd0930" containerName="istio-proxy" Oct 31 00:39:09.909357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.909341 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:09.961570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.961528 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:09.961886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:09.961866 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:39:10.014980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.014942 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8"] Oct 31 00:39:10.059547 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.059505 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.059728 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.059565 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcdw\" (UniqueName: \"kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.059728 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.059656 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.061759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.061740 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:10.061867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.061853 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:10.161617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.161530 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.161617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.161599 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcdw\" (UniqueName: \"kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.161846 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.161647 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.162437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.162418 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.164148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.164131 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.225026 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.224990 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcdw\" (UniqueName: \"kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw\") pod \"maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.522973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.522943 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:10.899262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:10.899231 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8"] Oct 31 00:39:11.015058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:11.015013 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" event={"ID":"ae9409dc-70ef-4b15-8bcb-133154bdc612","Type":"ContainerStarted","Data":"a61d4c4543673df90e2bac93aa5a3d8ddfce1b49a2be4f6dae60adc253aa07ff"} Oct 31 00:39:12.023386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:12.023352 5943 generic.go:356] "Generic (PLEG): container finished" podID="ae9409dc-70ef-4b15-8bcb-133154bdc612" containerID="de2c7ae8dec546085508dbd90fab8f69d7b373606d91462907e78af409093a94" exitCode=0 Oct 31 00:39:12.023817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:12.023449 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" event={"ID":"ae9409dc-70ef-4b15-8bcb-133154bdc612","Type":"ContainerDied","Data":"de2c7ae8dec546085508dbd90fab8f69d7b373606d91462907e78af409093a94"} Oct 31 00:39:13.189997 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.189969 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:13.299700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.299596 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca\") pod \"ae9409dc-70ef-4b15-8bcb-133154bdc612\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " Oct 31 00:39:13.299901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.299710 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw\" (UniqueName: \"kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw\") pod \"ae9409dc-70ef-4b15-8bcb-133154bdc612\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " Oct 31 00:39:13.299901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.299783 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca\") pod \"ae9409dc-70ef-4b15-8bcb-133154bdc612\" (UID: \"ae9409dc-70ef-4b15-8bcb-133154bdc612\") " Oct 31 00:39:13.300157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.300124 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca" pod "ae9409dc-70ef-4b15-8bcb-133154bdc612" (UID: "ae9409dc-70ef-4b15-8bcb-133154bdc612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:13.302049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.302020 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca" pod "ae9409dc-70ef-4b15-8bcb-133154bdc612" (UID: "ae9409dc-70ef-4b15-8bcb-133154bdc612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:13.302137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.302063 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw" pod "ae9409dc-70ef-4b15-8bcb-133154bdc612" (UID: "ae9409dc-70ef-4b15-8bcb-133154bdc612"). InnerVolumeSpecName "kube-api-access-cjcdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:13.401605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.401570 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:39:13.401605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.401600 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjcdw\" (UniqueName: \"kubernetes.io/projected/ae9409dc-70ef-4b15-8bcb-133154bdc612-kube-api-access-cjcdw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:39:13.401605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:13.401610 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9409dc-70ef-4b15-8bcb-133154bdc612-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:39:14.047644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:14.047616 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" Oct 31 00:39:14.051388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:14.051344 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8" event={"ID":"ae9409dc-70ef-4b15-8bcb-133154bdc612","Type":"ContainerDied","Data":"a61d4c4543673df90e2bac93aa5a3d8ddfce1b49a2be4f6dae60adc253aa07ff"} Oct 31 00:39:14.051548 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:14.051402 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61d4c4543673df90e2bac93aa5a3d8ddfce1b49a2be4f6dae60adc253aa07ff" Oct 31 00:39:14.457042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:14.456951 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:14.615910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:14.615869 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:19.689392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:19.689339 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:23.967077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:23.967038 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:39:24.501945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:24.501907 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:24.699748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:24.699713 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:24.845101 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:24.845066 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:25.848161 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:25.848122 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:26.046234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:26.046108 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:27.476586 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:39:27.476547 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:39:29.511502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:29.511466 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:34.514579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:34.514534 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.045451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:38.045416 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.537166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:38.537128 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:40.644461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:40.644426 5943 scope.go:122] "RemoveContainer" containerID="966f14f94eca4b39e4361d61595a76404431291cba458831bba965513089a3b9" Oct 31 00:39:41.288523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:41.288492 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:47.263405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.263369 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p"] Oct 31 00:39:47.263951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.263933 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ae9409dc-70ef-4b15-8bcb-133154bdc612" containerName="revoke-keys" Oct 31 00:39:47.264011 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.263957 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9409dc-70ef-4b15-8bcb-133154bdc612" containerName="revoke-keys" Oct 31 00:39:47.264115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.264104 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9409dc-70ef-4b15-8bcb-133154bdc612" containerName="revoke-keys" Oct 31 00:39:47.268918 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.268895 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.323386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.323335 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:47.326856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.326815 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p"] Oct 31 00:39:47.341934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.341902 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.342139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.341949 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.342139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.342102 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzx2\" (UniqueName: \"kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.372876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.372832 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:47.372876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.372832 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:39:47.373093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.372884 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:47.443609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.443568 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.443609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.443619 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.443886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.443709 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzx2\" (UniqueName: \"kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.444406 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.444381 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.446235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.446207 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.504891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.504859 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzx2\" (UniqueName: \"kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2\") pod \"maas-api-revoke-sub-ef70ff2f-qp47p\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.582525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.582422 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:47.889963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.889879 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:47.964864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:47.964783 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p"] Oct 31 00:39:48.044394 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:48.044360 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:48.253730 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:48.253699 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" event={"ID":"72688853-91e7-49ac-8133-da1bd472ade8","Type":"ContainerStarted","Data":"fd3657c40f73a16d5d0b9b82dbb0591b5e46901b37edb24e2d4bf1fe2cc8aea5"} Oct 31 00:39:48.253890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:48.253762 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" event={"ID":"72688853-91e7-49ac-8133-da1bd472ade8","Type":"ContainerStarted","Data":"82f64da577f4184be3a9ac03f9a7129c8126b6e0825e561d09e799f13d93b602"} Oct 31 00:39:48.276902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:48.276863 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:48.370545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:48.370474 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" podStartSLOduration=1.370458818 podStartE2EDuration="1.370458818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:48.36964163 +0000 UTC m=+2168.462931818" watchObservedRunningTime="2026-10-31 00:39:48.370458818 +0000 UTC m=+2168.463749001" Oct 31 00:39:49.261136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:49.261105 5943 generic.go:356] "Generic (PLEG): container finished" podID="72688853-91e7-49ac-8133-da1bd472ade8" containerID="fd3657c40f73a16d5d0b9b82dbb0591b5e46901b37edb24e2d4bf1fe2cc8aea5" exitCode=0 Oct 31 00:39:49.261343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:49.261225 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" event={"ID":"72688853-91e7-49ac-8133-da1bd472ade8","Type":"ContainerDied","Data":"fd3657c40f73a16d5d0b9b82dbb0591b5e46901b37edb24e2d4bf1fe2cc8aea5"} Oct 31 00:39:50.422681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.422652 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:50.425357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.425338 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:50.481371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.481331 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca\") pod \"72688853-91e7-49ac-8133-da1bd472ade8\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " Oct 31 00:39:50.481671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.481410 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2\" (UniqueName: \"kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2\") pod \"72688853-91e7-49ac-8133-da1bd472ade8\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " Oct 31 00:39:50.481671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.481494 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca\") pod \"72688853-91e7-49ac-8133-da1bd472ade8\" (UID: \"72688853-91e7-49ac-8133-da1bd472ade8\") " Oct 31 00:39:50.481950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.481920 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca" pod "72688853-91e7-49ac-8133-da1bd472ade8" (UID: "72688853-91e7-49ac-8133-da1bd472ade8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:50.483750 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.483730 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca" pod "72688853-91e7-49ac-8133-da1bd472ade8" (UID: "72688853-91e7-49ac-8133-da1bd472ade8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:50.483840 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.483756 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2" pod "72688853-91e7-49ac-8133-da1bd472ade8" (UID: "72688853-91e7-49ac-8133-da1bd472ade8"). InnerVolumeSpecName "kube-api-access-djzx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:50.583395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.583304 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.583395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.583337 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzx2\" (UniqueName: \"kubernetes.io/projected/72688853-91e7-49ac-8133-da1bd472ade8-kube-api-access-djzx2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.583395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:50.583347 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72688853-91e7-49ac-8133-da1bd472ade8-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:39:51.274591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:51.274558 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" Oct 31 00:39:51.274788 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:51.274563 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p" event={"ID":"72688853-91e7-49ac-8133-da1bd472ade8","Type":"ContainerDied","Data":"82f64da577f4184be3a9ac03f9a7129c8126b6e0825e561d09e799f13d93b602"} Oct 31 00:39:51.274788 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:51.274682 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82f64da577f4184be3a9ac03f9a7129c8126b6e0825e561d09e799f13d93b602" Oct 31 00:39:52.907522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:52.907467 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:57.477347 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:39:57.477308 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:39:57.701211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:57.701156 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:57.901715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:57.897447 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:58.045797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:39:58.045760 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:02.689396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:02.689357 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:06.465844 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:06.465792 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:06.620653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:06.620609 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:07.667819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:07.667790 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:12.642356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:12.642320 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:40:14.004734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:14.004700 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:14.160835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:14.160800 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:14.463993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:14.463945 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:18.975715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:18.975683 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:23.883730 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:23.883694 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:27.477809 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:40:27.477773 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:40:28.715042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:28.714999 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:33.710479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:33.710434 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:38.588746 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:38.588711 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:40.675593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:40.675552 5943 scope.go:122] "RemoveContainer" containerID="60dead7e78ba60393cd5e376c3dcc536cc11f0fcc508aaed94148588c4942304" Oct 31 00:40:40.685299 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:40.685273 5943 scope.go:122] "RemoveContainer" containerID="e8c057d953fe32d97320ee431edd487a0e32742874681e86e5858b72da825f04" Oct 31 00:40:43.467471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:43.467433 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:43.741564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:43.741468 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:40:45.663986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:45.663936 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:47.086004 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.085972 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c"] Oct 31 00:40:47.086470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.086457 5943 cpu_manager.go:414] "containerMap: removing container" podUID="72688853-91e7-49ac-8133-da1bd472ade8" containerName="revoke-keys" Oct 31 00:40:47.086521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.086471 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72688853-91e7-49ac-8133-da1bd472ade8" containerName="revoke-keys" Oct 31 00:40:47.086598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.086586 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="72688853-91e7-49ac-8133-da1bd472ade8" containerName="revoke-keys" Oct 31 00:40:47.091321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.091296 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.147233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.146979 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:47.147854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.147827 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:40:47.148576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.148553 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:40:47.150575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.150540 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c"] Oct 31 00:40:47.192612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.192580 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.192824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.192619 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.192824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.192778 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94d5h\" (UniqueName: \"kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.196649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.196625 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:40:47.293986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.293948 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.294220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.294070 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d5h\" (UniqueName: \"kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.294332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.294316 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.294912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.294885 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.296661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.296636 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.360546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.360473 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d5h\" (UniqueName: \"kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h\") pod \"maas-api-revoke-keys-e2e-api-a-752be8-6qv8c\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.405965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.405927 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:47.788801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:47.788776 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c"] Oct 31 00:40:47.791304 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:40:47.791278 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75bd595_369e_4156_8f89_95b6e22a24f9.slice/crio-a98e88684e72b5168e883d13e6b4b7ff775f39d3fb3d8570d993fec8796d2b7b WatchSource:0}: Error finding container a98e88684e72b5168e883d13e6b4b7ff775f39d3fb3d8570d993fec8796d2b7b: Status 404 returned error can't find the container with id a98e88684e72b5168e883d13e6b4b7ff775f39d3fb3d8570d993fec8796d2b7b Oct 31 00:40:48.686387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:48.686348 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" event={"ID":"c75bd595-369e-4156-8f89-95b6e22a24f9","Type":"ContainerStarted","Data":"c827c27b1a72f393613f2305b663cf5ea88121d7cf3322dab03c58e0304113e4"} Oct 31 00:40:48.686814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:48.686422 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" event={"ID":"c75bd595-369e-4156-8f89-95b6e22a24f9","Type":"ContainerStarted","Data":"a98e88684e72b5168e883d13e6b4b7ff775f39d3fb3d8570d993fec8796d2b7b"} Oct 31 00:40:48.815643 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:48.815582 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" podStartSLOduration=1.8155695440000001 podStartE2EDuration="1.815569544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:48.814737988 +0000 UTC m=+2228.908028194" watchObservedRunningTime="2026-10-31 00:40:48.815569544 +0000 UTC m=+2228.908859731" Oct 31 00:40:49.068651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:49.068595 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:49.693075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:49.693040 5943 generic.go:356] "Generic (PLEG): container finished" podID="c75bd595-369e-4156-8f89-95b6e22a24f9" containerID="c827c27b1a72f393613f2305b663cf5ea88121d7cf3322dab03c58e0304113e4" exitCode=0 Oct 31 00:40:49.693514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:49.693123 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" event={"ID":"c75bd595-369e-4156-8f89-95b6e22a24f9","Type":"ContainerDied","Data":"c827c27b1a72f393613f2305b663cf5ea88121d7cf3322dab03c58e0304113e4"} Oct 31 00:40:50.770305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:50.763223 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:50.853690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:50.853601 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:51.038141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.038112 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca\") pod \"c75bd595-369e-4156-8f89-95b6e22a24f9\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " Oct 31 00:40:51.038344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.038164 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h\" (UniqueName: \"kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h\") pod \"c75bd595-369e-4156-8f89-95b6e22a24f9\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " Oct 31 00:40:51.038344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.038259 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca\") pod \"c75bd595-369e-4156-8f89-95b6e22a24f9\" (UID: \"c75bd595-369e-4156-8f89-95b6e22a24f9\") " Oct 31 00:40:51.038619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.038595 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca" pod "c75bd595-369e-4156-8f89-95b6e22a24f9" (UID: "c75bd595-369e-4156-8f89-95b6e22a24f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:51.040542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.040521 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca" pod "c75bd595-369e-4156-8f89-95b6e22a24f9" (UID: "c75bd595-369e-4156-8f89-95b6e22a24f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:51.040623 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.040540 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h" pod "c75bd595-369e-4156-8f89-95b6e22a24f9" (UID: "c75bd595-369e-4156-8f89-95b6e22a24f9"). InnerVolumeSpecName "kube-api-access-94d5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:51.140068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.140028 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:51.140068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.140063 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94d5h\" (UniqueName: \"kubernetes.io/projected/c75bd595-369e-4156-8f89-95b6e22a24f9-kube-api-access-94d5h\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:51.140345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.140080 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75bd595-369e-4156-8f89-95b6e22a24f9-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:51.706395 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.706365 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" Oct 31 00:40:51.706580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.706395 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c" event={"ID":"c75bd595-369e-4156-8f89-95b6e22a24f9","Type":"ContainerDied","Data":"a98e88684e72b5168e883d13e6b4b7ff775f39d3fb3d8570d993fec8796d2b7b"} Oct 31 00:40:51.706580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:51.706442 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a98e88684e72b5168e883d13e6b4b7ff775f39d3fb3d8570d993fec8796d2b7b" Oct 31 00:40:52.263961 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:52.263925 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk"] Oct 31 00:40:52.264506 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:52.264192 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" containerID="cri-o://969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890" gracePeriod=30 Oct 31 00:40:56.280310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:56.280273 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:56.484930 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:56.484892 5943 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:40:56.485103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:56.484965 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:57.487666 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:40:57.487628 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:40:57.542976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.542903 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:40:57.611274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611240 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611282 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611370 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611401 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611445 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611492 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611528 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611580 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611633 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs\") pod \"b521294b-b616-441b-920f-4418e7855055\" (UID: \"b521294b-b616-441b-920f-4418e7855055\") " Oct 31 00:40:57.611929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611722 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:57.611929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611814 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:57.612040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.611967 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:57.612098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.612036 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:57.612347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.612322 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:57.612539 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.612393 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b521294b-b616-441b-920f-4418e7855055-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.612628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.612547 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.612628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.612569 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.612628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.612588 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.613922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.613899 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:57.614125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.614103 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:40:57.614396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.614380 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:57.614469 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.614415 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2" pod "b521294b-b616-441b-920f-4418e7855055" (UID: "b521294b-b616-441b-920f-4418e7855055"). InnerVolumeSpecName "kube-api-access-nbcw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:57.714051 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.714018 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.714051 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.714053 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b521294b-b616-441b-920f-4418e7855055-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.714282 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.714067 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.714282 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.714083 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbcw2\" (UniqueName: \"kubernetes.io/projected/b521294b-b616-441b-920f-4418e7855055-kube-api-access-nbcw2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.714282 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.714099 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b521294b-b616-441b-920f-4418e7855055-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:40:57.743417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.743379 5943 generic.go:356] "Generic (PLEG): container finished" podID="b521294b-b616-441b-920f-4418e7855055" containerID="969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890" exitCode=0 Oct 31 00:40:57.743626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.743443 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" Oct 31 00:40:57.743626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.743481 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" event={"ID":"b521294b-b616-441b-920f-4418e7855055","Type":"ContainerDied","Data":"969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890"} Oct 31 00:40:57.743626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.743565 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk" event={"ID":"b521294b-b616-441b-920f-4418e7855055","Type":"ContainerDied","Data":"3d0183144bddad0ec0a69112b9ab71c91aa40b151c3c4e20a49e6678031cfe4d"} Oct 31 00:40:57.743626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.743601 5943 scope.go:122] "RemoveContainer" containerID="969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890" Oct 31 00:40:57.754417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.754395 5943 scope.go:122] "RemoveContainer" containerID="969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890" Oct 31 00:40:57.754706 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:40:57.754683 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890\": container with ID starting with 969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890 not found: ID does not exist" containerID="969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890" Oct 31 00:40:57.754764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.754715 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890"} err="failed to get container status \"969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890\": rpc error: code = NotFound desc = could not find container \"969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890\": container with ID starting with 969f065a49b0c555d144bc5c84b1971f8b0cf4777611bcc7c6f3371b0daa5890 not found: ID does not exist" Oct 31 00:40:57.908486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.908405 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk"] Oct 31 00:40:57.914598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:57.914565 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-xbbwk"] Oct 31 00:40:58.041302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:58.041269 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b521294b-b616-441b-920f-4418e7855055" path="/var/lib/kubelet/pods/b521294b-b616-441b-920f-4418e7855055/volumes" Oct 31 00:40:58.983925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:58.983885 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:59.179971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:40:59.179909 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:03.046057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:03.046021 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:07.241039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:07.241004 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:07.792496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:07.792454 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:07.992996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:07.992961 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5"] Oct 31 00:41:07.993207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:07.993154 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" containerID="cri-o://a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d" gracePeriod=30 Oct 31 00:41:08.092553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:08.092458 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.108038 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:09.107992 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.375075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:09.374977 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.533519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:09.533481 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.814028 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:09.813998 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8"] Oct 31 00:41:09.860930 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:09.860896 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bba5ed-sqjj8"] Oct 31 00:41:10.043211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:10.043141 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9409dc-70ef-4b15-8bcb-133154bdc612" path="/var/lib/kubelet/pods/ae9409dc-70ef-4b15-8bcb-133154bdc612/volumes" Oct 31 00:41:10.252255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:10.252145 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:10.637612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:10.637575 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:41:10.637917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:10.637878 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:11.342043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:11.342006 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:12.674289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.674248 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn"] Oct 31 00:41:12.674969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.674948 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c75bd595-369e-4156-8f89-95b6e22a24f9" containerName="revoke-keys" Oct 31 00:41:12.675062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.674971 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75bd595-369e-4156-8f89-95b6e22a24f9" containerName="revoke-keys" Oct 31 00:41:12.675062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.675021 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" Oct 31 00:41:12.675062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.675029 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" Oct 31 00:41:12.675244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.675193 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75bd595-369e-4156-8f89-95b6e22a24f9" containerName="revoke-keys" Oct 31 00:41:12.675244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.675215 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b521294b-b616-441b-920f-4418e7855055" containerName="istio-proxy" Oct 31 00:41:12.679167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.679144 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.731912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.731877 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:41:12.731912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.731914 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:41:12.781474 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.781438 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:41:12.781690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.781485 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:41:12.783952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.783923 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn"] Oct 31 00:41:12.803555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.803523 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.803748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.803583 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.803748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.803726 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.904964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.904928 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.905158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.904996 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.905158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.905059 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.905707 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.905685 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.907586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.907567 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.967449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.967375 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr\") pod \"maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:12.995436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:12.995388 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:13.246526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.246502 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:41:13.385259 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:41:13.385229 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842986e3_a5e4_4eec_8567_8934d15fa75b.slice/crio-b178c0e9301ad35f8362a78085f10701deeb03a6abdd7779e892d312d3128e30 WatchSource:0}: Error finding container b178c0e9301ad35f8362a78085f10701deeb03a6abdd7779e892d312d3128e30: Status 404 returned error can't find the container with id b178c0e9301ad35f8362a78085f10701deeb03a6abdd7779e892d312d3128e30 Oct 31 00:41:13.411555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411524 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.411695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411576 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.411695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411632 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.411843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411725 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.411843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411779 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.411843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411813 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.412004 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411864 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.412004 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411916 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.412004 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.411961 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket\") pod \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\" (UID: \"39b6a45e-1e54-4170-81ed-65f7364ff9a6\") " Oct 31 00:41:13.412169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412011 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:13.412169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412033 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:13.412321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412291 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:13.412553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412513 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:13.412553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412518 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.412553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412540 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:13.412742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412559 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.412742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.412578 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.414038 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.414013 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:41:13.414544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.414519 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:13.414800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.414780 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "kube-api-access-6zx7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:13.414849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.414809 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token" pod "39b6a45e-1e54-4170-81ed-65f7364ff9a6" (UID: "39b6a45e-1e54-4170-81ed-65f7364ff9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:13.419669 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.419646 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn"] Oct 31 00:41:13.514117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.514086 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.514117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.514115 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.514354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.514126 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.514354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.514135 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zx7s\" (UniqueName: \"kubernetes.io/projected/39b6a45e-1e54-4170-81ed-65f7364ff9a6-kube-api-access-6zx7s\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.514354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.514146 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b6a45e-1e54-4170-81ed-65f7364ff9a6-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.514354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.514154 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b6a45e-1e54-4170-81ed-65f7364ff9a6-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:13.846641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.846555 5943 generic.go:356] "Generic (PLEG): container finished" podID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerID="a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d" exitCode=0 Oct 31 00:41:13.846641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.846609 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" Oct 31 00:41:13.847229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.846635 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" event={"ID":"39b6a45e-1e54-4170-81ed-65f7364ff9a6","Type":"ContainerDied","Data":"a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d"} Oct 31 00:41:13.847229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.846702 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5" event={"ID":"39b6a45e-1e54-4170-81ed-65f7364ff9a6","Type":"ContainerDied","Data":"5312d3eafa5aa3549874c10b0805e86b78c6e667297ec50aca33b7d1f1c55183"} Oct 31 00:41:13.847229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.846729 5943 scope.go:122] "RemoveContainer" containerID="a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d" Oct 31 00:41:13.848419 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.848378 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" event={"ID":"842986e3-a5e4-4eec-8567-8934d15fa75b","Type":"ContainerStarted","Data":"6c4f6b85b8494065c5fa736e4177ba43a117402c1c8fee3300e66edae88ff154"} Oct 31 00:41:13.848533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.848436 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" event={"ID":"842986e3-a5e4-4eec-8567-8934d15fa75b","Type":"ContainerStarted","Data":"b178c0e9301ad35f8362a78085f10701deeb03a6abdd7779e892d312d3128e30"} Oct 31 00:41:13.857145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.857118 5943 scope.go:122] "RemoveContainer" containerID="a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d" Oct 31 00:41:13.857456 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:41:13.857431 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d\": container with ID starting with a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d not found: ID does not exist" containerID="a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d" Oct 31 00:41:13.857531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:13.857464 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d"} err="failed to get container status \"a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d\": rpc error: code = NotFound desc = could not find container \"a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d\": container with ID starting with a215e43dac24b289792e9ca447ea3015b4c459dcc6c2406f64f87379d44d031d not found: ID does not exist" Oct 31 00:41:14.020326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.020268 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" podStartSLOduration=2.020251467 podStartE2EDuration="2.020251467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:14.017632728 +0000 UTC m=+2254.110922933" watchObservedRunningTime="2026-10-31 00:41:14.020251467 +0000 UTC m=+2254.113541655" Oct 31 00:41:14.131752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.131670 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5"] Oct 31 00:41:14.133793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.133769 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-989s5"] Oct 31 00:41:14.587641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.587608 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z"] Oct 31 00:41:14.587838 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.587797 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" containerID="cri-o://d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024" gracePeriod=30 Oct 31 00:41:14.647209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.647140 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:14.857985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.857894 5943 generic.go:356] "Generic (PLEG): container finished" podID="842986e3-a5e4-4eec-8567-8934d15fa75b" containerID="6c4f6b85b8494065c5fa736e4177ba43a117402c1c8fee3300e66edae88ff154" exitCode=0 Oct 31 00:41:14.857985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.857956 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" event={"ID":"842986e3-a5e4-4eec-8567-8934d15fa75b","Type":"ContainerDied","Data":"6c4f6b85b8494065c5fa736e4177ba43a117402c1c8fee3300e66edae88ff154"} Oct 31 00:41:14.949533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:14.949493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:15.101434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:15.101390 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:15.314902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:15.314864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:16.013564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.013537 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:16.042941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.042911 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" path="/var/lib/kubelet/pods/39b6a45e-1e54-4170-81ed-65f7364ff9a6/volumes" Oct 31 00:41:16.147316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.147234 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca\") pod \"842986e3-a5e4-4eec-8567-8934d15fa75b\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " Oct 31 00:41:16.147467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.147439 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr\") pod \"842986e3-a5e4-4eec-8567-8934d15fa75b\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " Oct 31 00:41:16.148116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.147565 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca\") pod \"842986e3-a5e4-4eec-8567-8934d15fa75b\" (UID: \"842986e3-a5e4-4eec-8567-8934d15fa75b\") " Oct 31 00:41:16.148116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.147696 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca" pod "842986e3-a5e4-4eec-8567-8934d15fa75b" (UID: "842986e3-a5e4-4eec-8567-8934d15fa75b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:16.148593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.148570 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:16.149920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.149891 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr" pod "842986e3-a5e4-4eec-8567-8934d15fa75b" (UID: "842986e3-a5e4-4eec-8567-8934d15fa75b"). InnerVolumeSpecName "kube-api-access-t4hnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:16.150063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.150041 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca" pod "842986e3-a5e4-4eec-8567-8934d15fa75b" (UID: "842986e3-a5e4-4eec-8567-8934d15fa75b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:16.249847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.249799 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/842986e3-a5e4-4eec-8567-8934d15fa75b-kube-api-access-t4hnr\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:16.250340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.250318 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842986e3-a5e4-4eec-8567-8934d15fa75b-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:16.872544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.872514 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" Oct 31 00:41:16.872544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.872527 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn" event={"ID":"842986e3-a5e4-4eec-8567-8934d15fa75b","Type":"ContainerDied","Data":"b178c0e9301ad35f8362a78085f10701deeb03a6abdd7779e892d312d3128e30"} Oct 31 00:41:16.872759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:16.872576 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b178c0e9301ad35f8362a78085f10701deeb03a6abdd7779e892d312d3128e30" Oct 31 00:41:17.891382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:17.891346 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:41:17.891788 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:17.891415 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:19.799222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.799098 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:19.881232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.881199 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:41:19.892131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.892098 5943 generic.go:356] "Generic (PLEG): container finished" podID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerID="d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024" exitCode=0 Oct 31 00:41:19.892290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.892144 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" Oct 31 00:41:19.892290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.892207 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" event={"ID":"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5","Type":"ContainerDied","Data":"d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024"} Oct 31 00:41:19.892290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.892262 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z" event={"ID":"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5","Type":"ContainerDied","Data":"8ccf9ddda0e065e2dbfae3ddab411352115634e85fbf8a35200faad885165e86"} Oct 31 00:41:19.892290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.892285 5943 scope.go:122] "RemoveContainer" containerID="d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024" Oct 31 00:41:19.901855 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.901835 5943 scope.go:122] "RemoveContainer" containerID="d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024" Oct 31 00:41:19.902152 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:41:19.902132 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024\": container with ID starting with d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024 not found: ID does not exist" containerID="d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024" Oct 31 00:41:19.902236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.902163 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024"} err="failed to get container status \"d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024\": rpc error: code = NotFound desc = could not find container \"d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024\": container with ID starting with d8fb1607300d281e40a6fab8a4b6235bb77bdd4a3eeb436b43ac7047f02f1024 not found: ID does not exist" Oct 31 00:41:19.911675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911653 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.911781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911702 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.911781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911739 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.911890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911849 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.912061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911913 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.912061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911974 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:19.912061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911989 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:19.912061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.911982 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.912061 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912044 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.912349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912163 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.912349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912217 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:19.912349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912236 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert\") pod \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\" (UID: \"d47ac0d7-d51b-4e0f-b17e-558e7ce33de5\") " Oct 31 00:41:19.912813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912737 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:19.912813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912761 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:19.912813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912778 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:19.912813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912757 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:19.913060 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.912821 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:19.914257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.914233 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:19.914369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.914350 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "kube-api-access-qkkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:19.914640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.914622 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:41:19.914712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.914675 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy" pod "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" (UID: "d47ac0d7-d51b-4e0f-b17e-558e7ce33de5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:19.984854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:19.984820 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:41:20.013931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.013899 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:20.013931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.013930 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:20.014206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.013942 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:20.014206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.013952 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:20.014206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.013963 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:20.014206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.013972 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkls\" (UniqueName: \"kubernetes.io/projected/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5-kube-api-access-qkkls\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:20.308316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.308268 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z"] Oct 31 00:41:20.310619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:20.310594 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-f7s5z"] Oct 31 00:41:21.044455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:21.044416 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:22.040885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:22.040852 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" path="/var/lib/kubelet/pods/d47ac0d7-d51b-4e0f-b17e-558e7ce33de5/volumes" Oct 31 00:41:22.318569 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:22.318476 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.003545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.003506 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp"] Oct 31 00:41:23.004093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004071 5943 cpu_manager.go:414] "containerMap: removing container" podUID="842986e3-a5e4-4eec-8567-8934d15fa75b" containerName="revoke-keys" Oct 31 00:41:23.004288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004258 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842986e3-a5e4-4eec-8567-8934d15fa75b" containerName="revoke-keys" Oct 31 00:41:23.004402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004299 5943 cpu_manager.go:414] "containerMap: removing container" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" Oct 31 00:41:23.004402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004306 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" Oct 31 00:41:23.004402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004322 5943 cpu_manager.go:414] "containerMap: removing container" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" Oct 31 00:41:23.004402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004327 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" Oct 31 00:41:23.004602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004464 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47ac0d7-d51b-4e0f-b17e-558e7ce33de5" containerName="istio-proxy" Oct 31 00:41:23.004602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004480 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="842986e3-a5e4-4eec-8567-8934d15fa75b" containerName="revoke-keys" Oct 31 00:41:23.004602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.004489 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b6a45e-1e54-4170-81ed-65f7364ff9a6" containerName="istio-proxy" Oct 31 00:41:23.009203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.009164 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.048706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.048657 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.048887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.048767 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.048952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.048902 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.049029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.048990 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.049227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.049167 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.049442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.049416 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.049551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.049461 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.049551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.049486 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.049665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.049547 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.067906 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.067875 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-868b06-openshift-default-dockercfg-kvdm4\"" Oct 31 00:41:23.107490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.107460 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.117100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.117068 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp"] Oct 31 00:41:23.151046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151004 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151076 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151105 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151133 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151165 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151212 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151240 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151372 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.151618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.151421 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.152827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.152164 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.152827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.152688 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.153076 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.152897 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.156910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.156882 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.157368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.157347 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.157770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.157738 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.170781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.170747 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.217922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.217829 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.218085 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.217960 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9\") pod \"e2e-praxis-b-868b06-openshift-default-58984b7645-274sp\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.295451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.295411 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.325476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.325438 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:23.776106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.776075 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp"] Oct 31 00:41:23.824830 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:41:23.824799 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6edfaa_bdcc_4657_9055_89773e6056ae.slice/crio-0cff27dbdd6cba6b5322926b165c45e7e497861618069467d2007b890f7e5b8f WatchSource:0}: Error finding container 0cff27dbdd6cba6b5322926b165c45e7e497861618069467d2007b890f7e5b8f: Status 404 returned error can't find the container with id 0cff27dbdd6cba6b5322926b165c45e7e497861618069467d2007b890f7e5b8f Oct 31 00:41:23.828084 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.828047 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:41:23.828216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.828137 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:41:23.828288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.828232 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:41:23.927365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.927312 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" event={"ID":"ea6edfaa-bdcc-4657-9055-89773e6056ae","Type":"ContainerStarted","Data":"015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e"} Oct 31 00:41:23.927568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:23.927393 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" event={"ID":"ea6edfaa-bdcc-4657-9055-89773e6056ae","Type":"ContainerStarted","Data":"0cff27dbdd6cba6b5322926b165c45e7e497861618069467d2007b890f7e5b8f"} Oct 31 00:41:24.053933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:24.051880 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podStartSLOduration=2.051861412 podStartE2EDuration="2.051861412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:24.051509274 +0000 UTC m=+2264.144799461" watchObservedRunningTime="2026-10-31 00:41:24.051861412 +0000 UTC m=+2264.145151600" Oct 31 00:41:24.326386 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:24.326299 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:24.332315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:24.332288 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:24.544455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:24.544420 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:24.934151 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:24.934115 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:24.935298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:24.935277 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:41:25.547711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:25.547662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:26.969298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:26.969266 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.338504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:27.338469 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.475318 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:41:27.475276 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:41:30.286323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.286280 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.519769 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.519712 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.641024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.640929 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q"] Oct 31 00:41:30.645261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.645235 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.699695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.699658 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-868b06\"" Oct 31 00:41:30.747421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.747388 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.747617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.747427 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2h8\" (UniqueName: \"kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8\") pod \"payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.750426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.750393 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q"] Oct 31 00:41:30.849849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.849800 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.851214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.851168 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2h8\" (UniqueName: \"kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8\") pod \"payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.851625 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.851104 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.919317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.919232 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2h8\" (UniqueName: \"kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8\") pod \"payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:30.959453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:30.959408 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:31.341138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:31.341101 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q"] Oct 31 00:41:31.343857 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:41:31.343819 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c74d794_8738_4f7e_b3b8_cbc07f6f48df.slice/crio-f4ee4c03ed0a28896d21069abe26c732aabce20ef4654e034d5973b9b7a3dcfb WatchSource:0}: Error finding container f4ee4c03ed0a28896d21069abe26c732aabce20ef4654e034d5973b9b7a3dcfb: Status 404 returned error can't find the container with id f4ee4c03ed0a28896d21069abe26c732aabce20ef4654e034d5973b9b7a3dcfb Oct 31 00:41:31.949333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:31.949298 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.978768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:31.978730 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" event={"ID":"4c74d794-8738-4f7e-b3b8-cbc07f6f48df","Type":"ContainerStarted","Data":"266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03"} Oct 31 00:41:31.978960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:31.978800 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:31.978960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:31.978823 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" event={"ID":"4c74d794-8738-4f7e-b3b8-cbc07f6f48df","Type":"ContainerStarted","Data":"f4ee4c03ed0a28896d21069abe26c732aabce20ef4654e034d5973b9b7a3dcfb"} Oct 31 00:41:32.098614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:32.098561 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" podStartSLOduration=2.098543406 podStartE2EDuration="2.098543406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:32.095851488 +0000 UTC m=+2272.189141677" watchObservedRunningTime="2026-10-31 00:41:32.098543406 +0000 UTC m=+2272.191833593" Oct 31 00:41:32.792729 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:32.792687 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:37.662063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:37.662022 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:40.716657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:40.716626 5943 scope.go:122] "RemoveContainer" containerID="9262fcf892ac1e0d1fa80623666f79e1fe846fbdd3c8efda98ff6abd4a3c2946" Oct 31 00:41:42.841537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:42.841480 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:42.988222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:42.988168 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:41:43.584357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:43.584305 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:43.793285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:43.793248 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:43.948851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:43.948759 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:47.670050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:47.670007 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:51.142851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:51.142819 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:51.939158 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:51.939107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:52.686857 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:52.686821 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:53.105389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:53.105351 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:53.329750 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:53.329710 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z"] Oct 31 00:41:53.330015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:53.329989 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" containerID="cri-o://3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9" gracePeriod=30 Oct 31 00:41:57.476107 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:41:57.476062 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:41:57.591167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:57.591121 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:58.593643 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.593613 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:41:58.678576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678487 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678538 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678574 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678620 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678705 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678759 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678798 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.678897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678880 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.679153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678912 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data\") pod \"84585842-bbef-4602-bf0d-b6e9cd15d756\" (UID: \"84585842-bbef-4602-bf0d-b6e9cd15d756\") " Oct 31 00:41:58.679153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.678960 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:58.679153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679007 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:58.679153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679060 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:58.679627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679381 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:58.679627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679387 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:58.679922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679899 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.680013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679927 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.680013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679943 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.680013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679958 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84585842-bbef-4602-bf0d-b6e9cd15d756-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.680013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.679973 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.681257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.681233 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:41:58.681345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.681264 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:58.681398 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.681360 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:58.681462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.681443 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s" pod "84585842-bbef-4602-bf0d-b6e9cd15d756" (UID: "84585842-bbef-4602-bf0d-b6e9cd15d756"). InnerVolumeSpecName "kube-api-access-4sw5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:58.780902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.780871 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.780902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.780898 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.781104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.780909 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84585842-bbef-4602-bf0d-b6e9cd15d756-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:58.781104 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:58.780921 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sw5s\" (UniqueName: \"kubernetes.io/projected/84585842-bbef-4602-bf0d-b6e9cd15d756-kube-api-access-4sw5s\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:59.143359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.143320 5943 generic.go:356] "Generic (PLEG): container finished" podID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerID="3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9" exitCode=0 Oct 31 00:41:59.143572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.143371 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" Oct 31 00:41:59.143572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.143397 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" event={"ID":"84585842-bbef-4602-bf0d-b6e9cd15d756","Type":"ContainerDied","Data":"3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9"} Oct 31 00:41:59.143572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.143448 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z" event={"ID":"84585842-bbef-4602-bf0d-b6e9cd15d756","Type":"ContainerDied","Data":"62ba0c0116d9f46d736144f68714ee10b7b7a9d148789034ec661f6fc4474f47"} Oct 31 00:41:59.143572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.143469 5943 scope.go:122] "RemoveContainer" containerID="3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9" Oct 31 00:41:59.155126 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.155107 5943 scope.go:122] "RemoveContainer" containerID="3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9" Oct 31 00:41:59.155461 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:41:59.155441 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9\": container with ID starting with 3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9 not found: ID does not exist" containerID="3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9" Oct 31 00:41:59.155537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.155469 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9"} err="failed to get container status \"3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9\": rpc error: code = NotFound desc = could not find container \"3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9\": container with ID starting with 3331cfaafab8ff195bb7a4e67f38743080fa215d84832098b5653082408182e9 not found: ID does not exist" Oct 31 00:41:59.290578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.290540 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:59.330139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.330106 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z"] Oct 31 00:41:59.333442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.333410 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7v26z"] Oct 31 00:41:59.627760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.627727 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:59.826068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:41:59.826025 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:00.041219 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:00.041166 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" path="/var/lib/kubelet/pods/84585842-bbef-4602-bf0d-b6e9cd15d756/volumes" Oct 31 00:42:01.157606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.157571 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll"] Oct 31 00:42:01.158333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.158311 5943 cpu_manager.go:414] "containerMap: removing container" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" Oct 31 00:42:01.158456 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.158334 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" Oct 31 00:42:01.158520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.158486 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="84585842-bbef-4602-bf0d-b6e9cd15d756" containerName="istio-proxy" Oct 31 00:42:01.163146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.163125 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.213074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.213042 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.213074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.213075 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.213366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.213343 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glf8w\" (UniqueName: \"kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.217146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.217118 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:42:01.266553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.266518 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:01.266553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.266527 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:42:01.266771 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.266647 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:42:01.268894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.268871 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll"] Oct 31 00:42:01.314708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.314664 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.314878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.314736 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.315037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.315016 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glf8w\" (UniqueName: \"kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.315410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.315384 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.317331 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.317308 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.376555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.376525 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glf8w\" (UniqueName: \"kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w\") pod \"maas-api-revoke-keys-e2e-ipp-b-70babe-62wll\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.475939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.475851 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:01.854444 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.854421 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll"] Oct 31 00:42:01.856284 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:42:01.856255 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679e55e0_be63_4c8d_89d5_9a86b29e47e1.slice/crio-ae5688aba2d729ab306e2d4749c512bd234148a17e55d353b3f3032dfa1171b8 WatchSource:0}: Error finding container ae5688aba2d729ab306e2d4749c512bd234148a17e55d353b3f3032dfa1171b8: Status 404 returned error can't find the container with id ae5688aba2d729ab306e2d4749c512bd234148a17e55d353b3f3032dfa1171b8 Oct 31 00:42:01.989315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:01.989282 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:02.167842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:02.167754 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" event={"ID":"679e55e0-be63-4c8d-89d5-9a86b29e47e1","Type":"ContainerStarted","Data":"ae5688aba2d729ab306e2d4749c512bd234148a17e55d353b3f3032dfa1171b8"} Oct 31 00:42:02.565220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:02.563621 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:02.714157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:02.714122 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:03.175851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:03.175758 5943 generic.go:356] "Generic (PLEG): container finished" podID="679e55e0-be63-4c8d-89d5-9a86b29e47e1" containerID="a01290c79d3f1141cc4999ccba30f4b37102131c668050ac9ddbde628c310372" exitCode=0 Oct 31 00:42:03.176266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:03.175849 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" event={"ID":"679e55e0-be63-4c8d-89d5-9a86b29e47e1","Type":"ContainerDied","Data":"a01290c79d3f1141cc4999ccba30f4b37102131c668050ac9ddbde628c310372"} Oct 31 00:42:03.711686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:03.711651 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:04.048160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.048124 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:04.195706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.195650 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:04.218800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.218767 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn"] Oct 31 00:42:04.222762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.222734 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-97fc39-586xn"] Oct 31 00:42:04.358074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.358048 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:04.457106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.457072 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca\") pod \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " Oct 31 00:42:04.457355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.457141 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca\") pod \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " Oct 31 00:42:04.457355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.457170 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w\" (UniqueName: \"kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w\") pod \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\" (UID: \"679e55e0-be63-4c8d-89d5-9a86b29e47e1\") " Oct 31 00:42:04.457538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.457515 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca" pod "679e55e0-be63-4c8d-89d5-9a86b29e47e1" (UID: "679e55e0-be63-4c8d-89d5-9a86b29e47e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:04.457715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.457700 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:04.459480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.459453 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w" pod "679e55e0-be63-4c8d-89d5-9a86b29e47e1" (UID: "679e55e0-be63-4c8d-89d5-9a86b29e47e1"). InnerVolumeSpecName "kube-api-access-glf8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:04.459597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.459472 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca" pod "679e55e0-be63-4c8d-89d5-9a86b29e47e1" (UID: "679e55e0-be63-4c8d-89d5-9a86b29e47e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:04.558894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.558856 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679e55e0-be63-4c8d-89d5-9a86b29e47e1-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:04.558894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.558891 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glf8w\" (UniqueName: \"kubernetes.io/projected/679e55e0-be63-4c8d-89d5-9a86b29e47e1-kube-api-access-glf8w\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:04.697314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:04.697225 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:05.188977 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:05.188942 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" Oct 31 00:42:05.189153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:05.188972 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll" event={"ID":"679e55e0-be63-4c8d-89d5-9a86b29e47e1","Type":"ContainerDied","Data":"ae5688aba2d729ab306e2d4749c512bd234148a17e55d353b3f3032dfa1171b8"} Oct 31 00:42:05.189153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:05.189025 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae5688aba2d729ab306e2d4749c512bd234148a17e55d353b3f3032dfa1171b8" Oct 31 00:42:06.040823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:06.040789 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842986e3-a5e4-4eec-8567-8934d15fa75b" path="/var/lib/kubelet/pods/842986e3-a5e4-4eec-8567-8934d15fa75b/volumes" Oct 31 00:42:10.658847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:10.658813 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:10.861435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:10.861396 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:11.088319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:11.088280 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:11.337439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:11.337402 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:14.958167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:14.958129 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:15.490905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:15.490868 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.127451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.127413 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg"] Oct 31 00:42:17.127980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.127963 5943 cpu_manager.go:414] "containerMap: removing container" podUID="679e55e0-be63-4c8d-89d5-9a86b29e47e1" containerName="revoke-keys" Oct 31 00:42:17.128024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.127982 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679e55e0-be63-4c8d-89d5-9a86b29e47e1" containerName="revoke-keys" Oct 31 00:42:17.128082 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.128072 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="679e55e0-be63-4c8d-89d5-9a86b29e47e1" containerName="revoke-keys" Oct 31 00:42:17.132613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.132594 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.188568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.188531 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:17.188781 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.188665 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:42:17.238168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.238140 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:42:17.238168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.238148 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:42:17.240628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.240608 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg"] Oct 31 00:42:17.311933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.311899 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.312115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.311967 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nc69\" (UniqueName: \"kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.312115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.312093 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.413276 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.413171 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.413453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.413329 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.413453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.413378 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nc69\" (UniqueName: \"kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.414062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.414035 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.415888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.415861 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.474760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.474732 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nc69\" (UniqueName: \"kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:17.746692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:17.746528 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:18.127140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:18.127118 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg"] Oct 31 00:42:18.129833 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:42:18.129801 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dd3647_3cc7_4417_8e67_cafd004a8df0.slice/crio-5de2eff96cd06b7cfe752414d8c1e91e461c4ea59ad360aac322246abf6bcc12 WatchSource:0}: Error finding container 5de2eff96cd06b7cfe752414d8c1e91e461c4ea59ad360aac322246abf6bcc12: Status 404 returned error can't find the container with id 5de2eff96cd06b7cfe752414d8c1e91e461c4ea59ad360aac322246abf6bcc12 Oct 31 00:42:18.197147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:18.197121 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:18.269513 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:18.269478 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" event={"ID":"98dd3647-3cc7-4417-8e67-cafd004a8df0","Type":"ContainerStarted","Data":"5de2eff96cd06b7cfe752414d8c1e91e461c4ea59ad360aac322246abf6bcc12"} Oct 31 00:42:19.277253 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:19.277221 5943 generic.go:356] "Generic (PLEG): container finished" podID="98dd3647-3cc7-4417-8e67-cafd004a8df0" containerID="10691af3a1c1eb1aade603dedc413b1d336f79025fa1876b2435b724b4cf7211" exitCode=0 Oct 31 00:42:19.277655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:19.277262 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" event={"ID":"98dd3647-3cc7-4417-8e67-cafd004a8df0","Type":"ContainerDied","Data":"10691af3a1c1eb1aade603dedc413b1d336f79025fa1876b2435b724b4cf7211"} Oct 31 00:42:20.427576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.427552 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:20.447452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.447423 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca\") pod \"98dd3647-3cc7-4417-8e67-cafd004a8df0\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " Oct 31 00:42:20.447618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.447483 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69\" (UniqueName: \"kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69\") pod \"98dd3647-3cc7-4417-8e67-cafd004a8df0\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " Oct 31 00:42:20.447618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.447567 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca\") pod \"98dd3647-3cc7-4417-8e67-cafd004a8df0\" (UID: \"98dd3647-3cc7-4417-8e67-cafd004a8df0\") " Oct 31 00:42:20.447772 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.447751 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca" pod "98dd3647-3cc7-4417-8e67-cafd004a8df0" (UID: "98dd3647-3cc7-4417-8e67-cafd004a8df0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:20.447845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.447830 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.449733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.449709 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69" pod "98dd3647-3cc7-4417-8e67-cafd004a8df0" (UID: "98dd3647-3cc7-4417-8e67-cafd004a8df0"). InnerVolumeSpecName "kube-api-access-2nc69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:20.449814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.449714 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca" pod "98dd3647-3cc7-4417-8e67-cafd004a8df0" (UID: "98dd3647-3cc7-4417-8e67-cafd004a8df0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:20.548893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.548821 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nc69\" (UniqueName: \"kubernetes.io/projected/98dd3647-3cc7-4417-8e67-cafd004a8df0-kube-api-access-2nc69\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:20.548893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:20.548848 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dd3647-3cc7-4417-8e67-cafd004a8df0-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.290327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:21.290297 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" Oct 31 00:42:21.290505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:21.290324 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg" event={"ID":"98dd3647-3cc7-4417-8e67-cafd004a8df0","Type":"ContainerDied","Data":"5de2eff96cd06b7cfe752414d8c1e91e461c4ea59ad360aac322246abf6bcc12"} Oct 31 00:42:21.290505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:21.290378 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de2eff96cd06b7cfe752414d8c1e91e461c4ea59ad360aac322246abf6bcc12" Oct 31 00:42:23.243366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:23.243332 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:23.963647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:23.963614 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:42:27.477421 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:42:27.477382 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:42:27.969904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:27.969863 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:32.831864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:32.831819 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.029797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:33.029756 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:37.086621 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:37.086577 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:37.248141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:37.248102 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:37.769442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:37.769401 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:37.928573 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:37.928530 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:38.864443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:38.864406 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:39.254748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:39.254662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:39.400315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:39.399129 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll"] Oct 31 00:42:39.434130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:39.434092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:39.446853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:39.446824 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-70babe-62wll"] Oct 31 00:42:39.922963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:39.922931 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:40.040421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:40.040389 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679e55e0-be63-4c8d-89d5-9a86b29e47e1" path="/var/lib/kubelet/pods/679e55e0-be63-4c8d-89d5-9a86b29e47e1/volumes" Oct 31 00:42:40.754007 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:40.753971 5943 scope.go:122] "RemoveContainer" containerID="54fdae4924d5cc4ba35ecf60a24974c26c5196ff71966e2865a839fdd45e56f1" Oct 31 00:42:43.343245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:43.343209 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:43.828727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:43.828696 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:46.818525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:46.818483 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:46.974040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:46.973995 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:47.213060 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:47.212958 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:48.790990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:48.790951 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:49.815776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:49.815735 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:50.076221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:50.076097 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:51.480369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:51.480338 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:52.916122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:52.916089 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:54.337760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:54.337671 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:54.512801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:54.512768 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:54.690740 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:54.690650 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:56.300576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:56.300541 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.484220 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:42:57.484165 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:42:57.653363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:57.652792 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.803072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:57.803041 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:58.007017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:58.006981 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:58.317066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:58.317005 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:58.523079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:58.523048 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:58.630912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:58.630824 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c"] Oct 31 00:42:58.679343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:58.679310 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-752be8-6qv8c"] Oct 31 00:42:59.073832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:42:59.073793 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:00.041740 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:00.041705 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75bd595-369e-4156-8f89-95b6e22a24f9" path="/var/lib/kubelet/pods/c75bd595-369e-4156-8f89-95b6e22a24f9/volumes" Oct 31 00:43:02.547614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:02.547579 5943 generic.go:356] "Generic (PLEG): container finished" podID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerID="015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e" exitCode=0 Oct 31 00:43:02.548162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:02.547660 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" event={"ID":"ea6edfaa-bdcc-4657-9055-89773e6056ae","Type":"ContainerDied","Data":"015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e"} Oct 31 00:43:02.548162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:02.548115 5943 scope.go:122] "RemoveContainer" containerID="015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e" Oct 31 00:43:02.549718 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:02.549683 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:43:02.549814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:02.549775 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:43:02.549866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:02.549835 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:43:03.181310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.181273 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m"] Oct 31 00:43:03.181779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.181767 5943 cpu_manager.go:414] "containerMap: removing container" podUID="98dd3647-3cc7-4417-8e67-cafd004a8df0" containerName="revoke-keys" Oct 31 00:43:03.181827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.181782 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dd3647-3cc7-4417-8e67-cafd004a8df0" containerName="revoke-keys" Oct 31 00:43:03.181887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.181878 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dd3647-3cc7-4417-8e67-cafd004a8df0" containerName="revoke-keys" Oct 31 00:43:03.184925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.184908 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.259458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.259418 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m"] Oct 31 00:43:03.259892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.259869 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:03.260112 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.260077 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:03.260324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.260307 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:03.261226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.260772 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.261226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.260858 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pxw\" (UniqueName: \"kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.261226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.261008 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.261426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.261340 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:43:03.362349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.362311 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.362565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.362383 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.362565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.362431 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pxw\" (UniqueName: \"kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.363115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.363092 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.365127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.365107 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.423311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.423280 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pxw\" (UniqueName: \"kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw\") pod \"maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.497993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.497913 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:03.562446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.562408 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" event={"ID":"ea6edfaa-bdcc-4657-9055-89773e6056ae","Type":"ContainerStarted","Data":"ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8"} Oct 31 00:43:03.792745 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.792715 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:03.899679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:03.899655 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m"] Oct 31 00:43:03.901813 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:43:03.901790 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724cdfa3_dda9_4ce3_864b_654139ed3e49.slice/crio-651486ce9b08bcac994db20f61083c9ee0e97149968113856587bc0fb2be88e1 WatchSource:0}: Error finding container 651486ce9b08bcac994db20f61083c9ee0e97149968113856587bc0fb2be88e1: Status 404 returned error can't find the container with id 651486ce9b08bcac994db20f61083c9ee0e97149968113856587bc0fb2be88e1 Oct 31 00:43:04.325996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:04.325956 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:43:04.326260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:04.326153 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:43:04.326260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:04.326228 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 00:43:04.569793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:04.569751 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" event={"ID":"724cdfa3-dda9-4ce3-864b-654139ed3e49","Type":"ContainerStarted","Data":"44ffd1de48bf5b985767278683298abc534da4827e3825d93e5e95c0bca021b7"} Oct 31 00:43:04.570234 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:04.569816 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" event={"ID":"724cdfa3-dda9-4ce3-864b-654139ed3e49","Type":"ContainerStarted","Data":"651486ce9b08bcac994db20f61083c9ee0e97149968113856587bc0fb2be88e1"} Oct 31 00:43:04.722491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:04.722433 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" podStartSLOduration=1.722415248 podStartE2EDuration="1.722415248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:04.71900605 +0000 UTC m=+2364.812296256" watchObservedRunningTime="2026-10-31 00:43:04.722415248 +0000 UTC m=+2364.815705435" Oct 31 00:43:05.326509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:05.326470 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:43:05.326695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:05.326533 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 00:43:05.577998 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:05.577965 5943 generic.go:356] "Generic (PLEG): container finished" podID="724cdfa3-dda9-4ce3-864b-654139ed3e49" containerID="44ffd1de48bf5b985767278683298abc534da4827e3825d93e5e95c0bca021b7" exitCode=0 Oct 31 00:43:05.578432 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:05.578047 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" event={"ID":"724cdfa3-dda9-4ce3-864b-654139ed3e49","Type":"ContainerDied","Data":"44ffd1de48bf5b985767278683298abc534da4827e3825d93e5e95c0bca021b7"} Oct 31 00:43:05.868951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:05.868854 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:06.054953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.054913 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:06.243714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.243628 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:06.325768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.325738 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:43:06.325987 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.325787 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 00:43:06.783277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.783252 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:06.906544 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.906463 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca\") pod \"724cdfa3-dda9-4ce3-864b-654139ed3e49\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " Oct 31 00:43:06.906700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.906663 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw\" (UniqueName: \"kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw\") pod \"724cdfa3-dda9-4ce3-864b-654139ed3e49\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " Oct 31 00:43:06.906700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.906689 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca\") pod \"724cdfa3-dda9-4ce3-864b-654139ed3e49\" (UID: \"724cdfa3-dda9-4ce3-864b-654139ed3e49\") " Oct 31 00:43:06.907108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.907083 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca" pod "724cdfa3-dda9-4ce3-864b-654139ed3e49" (UID: "724cdfa3-dda9-4ce3-864b-654139ed3e49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:06.908710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.908688 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca" pod "724cdfa3-dda9-4ce3-864b-654139ed3e49" (UID: "724cdfa3-dda9-4ce3-864b-654139ed3e49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:06.908790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:06.908750 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw" pod "724cdfa3-dda9-4ce3-864b-654139ed3e49" (UID: "724cdfa3-dda9-4ce3-864b-654139ed3e49"). InnerVolumeSpecName "kube-api-access-c2pxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:07.008620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.008551 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2pxw\" (UniqueName: \"kubernetes.io/projected/724cdfa3-dda9-4ce3-864b-654139ed3e49-kube-api-access-c2pxw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:07.008620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.008613 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:07.008885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.008628 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724cdfa3-dda9-4ce3-864b-654139ed3e49-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:07.106893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.106853 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h"] Oct 31 00:43:07.107461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.107439 5943 cpu_manager.go:414] "containerMap: removing container" podUID="724cdfa3-dda9-4ce3-864b-654139ed3e49" containerName="revoke-keys" Oct 31 00:43:07.107461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.107461 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724cdfa3-dda9-4ce3-864b-654139ed3e49" containerName="revoke-keys" Oct 31 00:43:07.107631 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.107576 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="724cdfa3-dda9-4ce3-864b-654139ed3e49" containerName="revoke-keys" Oct 31 00:43:07.110968 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.110946 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.181948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.181867 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h"] Oct 31 00:43:07.210934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.210900 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92pv\" (UniqueName: \"kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.211162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.211136 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.211281 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.211260 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.312546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.312509 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.312765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.312582 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.312765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.312658 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92pv\" (UniqueName: \"kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.313245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.313215 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.315280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.315254 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.326405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.326378 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:43:07.326540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.326443 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 00:43:07.374017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.373982 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92pv\" (UniqueName: \"kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv\") pod \"maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.424729 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.424697 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:07.590583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.590553 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" Oct 31 00:43:07.590754 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.590552 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m" event={"ID":"724cdfa3-dda9-4ce3-864b-654139ed3e49","Type":"ContainerDied","Data":"651486ce9b08bcac994db20f61083c9ee0e97149968113856587bc0fb2be88e1"} Oct 31 00:43:07.590754 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.590665 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="651486ce9b08bcac994db20f61083c9ee0e97149968113856587bc0fb2be88e1" Oct 31 00:43:07.817431 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:07.817404 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h"] Oct 31 00:43:08.325839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:08.325809 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:43:08.326037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:08.325864 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 00:43:08.599829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:08.599731 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" event={"ID":"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb","Type":"ContainerStarted","Data":"8f79dba6316f825719d6887de869e1a22c4713138169ee1cb04011499aa5d3af"} Oct 31 00:43:08.599829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:08.599788 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" event={"ID":"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb","Type":"ContainerStarted","Data":"f2ebfb95541eb143aa047f74d5f7ecd0c91d39724838cbfba1967789c4fb7424"} Oct 31 00:43:08.717554 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:08.717500 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" podStartSLOduration=1.7174850579999998 podStartE2EDuration="1.717485058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:08.714538792 +0000 UTC m=+2368.807828980" watchObservedRunningTime="2026-10-31 00:43:08.717485058 +0000 UTC m=+2368.810775244" Oct 31 00:43:09.097672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:09.097635 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:09.331079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:09.331045 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:43:09.331468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:09.331449 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:43:09.332110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:09.332091 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:43:09.621584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:09.621486 5943 generic.go:356] "Generic (PLEG): container finished" podID="c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" containerID="8f79dba6316f825719d6887de869e1a22c4713138169ee1cb04011499aa5d3af" exitCode=0 Oct 31 00:43:09.621751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:09.621570 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" event={"ID":"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb","Type":"ContainerDied","Data":"8f79dba6316f825719d6887de869e1a22c4713138169ee1cb04011499aa5d3af"} Oct 31 00:43:10.783952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.783928 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:10.854066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.854035 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca\") pod \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " Oct 31 00:43:10.854287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.854222 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv\" (UniqueName: \"kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv\") pod \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " Oct 31 00:43:10.854378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.854355 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca\") pod \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\" (UID: \"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb\") " Oct 31 00:43:10.854733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.854711 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca" pod "c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" (UID: "c1b82f48-1f68-4c8d-9688-da19d3b3e9fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:10.856717 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.856672 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv" pod "c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" (UID: "c1b82f48-1f68-4c8d-9688-da19d3b3e9fb"). InnerVolumeSpecName "kube-api-access-g92pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:10.856806 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.856775 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca" pod "c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" (UID: "c1b82f48-1f68-4c8d-9688-da19d3b3e9fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:10.955436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.955341 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:10.955436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.955375 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:10.955436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:10.955389 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g92pv\" (UniqueName: \"kubernetes.io/projected/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb-kube-api-access-g92pv\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:11.637706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:11.637673 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" Oct 31 00:43:11.637905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:11.637674 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h" event={"ID":"c1b82f48-1f68-4c8d-9688-da19d3b3e9fb","Type":"ContainerDied","Data":"f2ebfb95541eb143aa047f74d5f7ecd0c91d39724838cbfba1967789c4fb7424"} Oct 31 00:43:11.637905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:11.637797 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ebfb95541eb143aa047f74d5f7ecd0c91d39724838cbfba1967789c4fb7424" Oct 31 00:43:13.743453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:13.743408 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:14.440708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:14.440675 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:14.590439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:14.590399 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.143767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:15.143706 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.587653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:15.587610 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.789128 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:15.789087 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.964093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:15.963997 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.059850 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:16.059817 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m"] Oct 31 00:43:16.111583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:16.111550 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f6a8a5-6qp6m"] Oct 31 00:43:16.504481 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:16.504438 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.832381 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:16.832345 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.171562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:17.171464 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.336305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:17.336272 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.419604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:17.419566 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg"] Oct 31 00:43:17.468549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:17.468462 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c02a9-5jxvg"] Oct 31 00:43:17.876887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:17.876851 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.041035 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:18.041003 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724cdfa3-dda9-4ce3-864b-654139ed3e49" path="/var/lib/kubelet/pods/724cdfa3-dda9-4ce3-864b-654139ed3e49/volumes" Oct 31 00:43:18.041470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:18.041456 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dd3647-3cc7-4417-8e67-cafd004a8df0" path="/var/lib/kubelet/pods/98dd3647-3cc7-4417-8e67-cafd004a8df0/volumes" Oct 31 00:43:19.277298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:19.277261 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.371119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:19.371087 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:43:19.438920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:19.438882 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.367334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:22.367298 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.680904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:22.680820 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.939325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:22.939239 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.782907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.782861 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x"] Oct 31 00:43:23.783581 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.783562 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" containerName="revoke-keys" Oct 31 00:43:23.783642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.783583 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" containerName="revoke-keys" Oct 31 00:43:23.783700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.783657 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" containerName="revoke-keys" Oct 31 00:43:23.788249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.788228 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:23.840378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.840345 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:43:23.846116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.846098 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:23.890054 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.890024 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:23.890226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.890093 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:23.892487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.892465 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x"] Oct 31 00:43:23.925692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.925662 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:23.925896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.925873 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:23.925987 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:23.925969 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwnf\" (UniqueName: \"kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.027212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.027143 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.027420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.027255 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwnf\" (UniqueName: \"kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.027420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.027398 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.028005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.027976 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.029698 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.029680 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.097808 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.097735 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwnf\" (UniqueName: \"kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.101575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.101540 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:24.489677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.489236 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x"] Oct 31 00:43:24.724899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:24.724859 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" event={"ID":"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed","Type":"ContainerStarted","Data":"e1db06cfa7044268ac0fa582a582daf1021f7ee8001f56d72a4a44a1bbb4b954"} Oct 31 00:43:25.372870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:25.372828 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.560467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:25.560425 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.733366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:25.733286 5943 generic.go:356] "Generic (PLEG): container finished" podID="c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" containerID="e4b77e4bd800c4638ec278dc510674fa44b3776117f537148d75ba12af3e3819" exitCode=0 Oct 31 00:43:25.733510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:25.733369 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" event={"ID":"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed","Type":"ContainerDied","Data":"e4b77e4bd800c4638ec278dc510674fa44b3776117f537148d75ba12af3e3819"} Oct 31 00:43:25.802756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:25.802727 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.895371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.895345 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:26.960902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.960868 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca\") pod \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " Oct 31 00:43:26.961086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.960914 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf\" (UniqueName: \"kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf\") pod \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " Oct 31 00:43:26.961086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.960951 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca\") pod \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\" (UID: \"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed\") " Oct 31 00:43:26.961290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.961268 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca" pod "c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" (UID: "c0a68b40-0ff9-4359-83dd-2b5b04a8fbed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:26.961521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.961505 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:26.963340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.963317 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca" pod "c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" (UID: "c0a68b40-0ff9-4359-83dd-2b5b04a8fbed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:26.963433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:26.963344 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf" pod "c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" (UID: "c0a68b40-0ff9-4359-83dd-2b5b04a8fbed"). InnerVolumeSpecName "kube-api-access-nnwnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:27.062531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.062493 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwnf\" (UniqueName: \"kubernetes.io/projected/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-kube-api-access-nnwnf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:27.062531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.062523 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:27.401861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.401758 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:27.475846 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:43:27.475807 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:43:27.749001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.748909 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" Oct 31 00:43:27.749001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.748924 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x" event={"ID":"c0a68b40-0ff9-4359-83dd-2b5b04a8fbed","Type":"ContainerDied","Data":"e1db06cfa7044268ac0fa582a582daf1021f7ee8001f56d72a4a44a1bbb4b954"} Oct 31 00:43:27.749001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.748993 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1db06cfa7044268ac0fa582a582daf1021f7ee8001f56d72a4a44a1bbb4b954" Oct 31 00:43:27.871782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:27.871743 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:28.084290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:28.084229 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:28.340215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:28.340108 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:32.682723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:32.682685 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:32.875590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:32.875547 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.117664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:33.117628 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.316636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:33.316601 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.069547 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:36.069502 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.270140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:36.270106 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:37.936933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:37.936898 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:38.051879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:38.051843 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz"] Oct 31 00:43:38.104299 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:38.104248 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc33e50-w4zqz"] Oct 31 00:43:38.523528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:38.523491 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:38.730440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:38.730399 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:39.780083 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:39.780048 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.001804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.001768 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.041033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.040954 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64" path="/var/lib/kubelet/pods/2a0a400d-a0ff-4e7c-8e0c-88c68ff8ac64/volumes" Oct 31 00:43:40.156319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.156280 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.281950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.281907 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x"] Oct 31 00:43:40.331427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.331340 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a4f30-2lj7x"] Oct 31 00:43:40.367899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.367867 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.754030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:40.753947 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.041441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:42.041410 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" path="/var/lib/kubelet/pods/c0a68b40-0ff9-4359-83dd-2b5b04a8fbed/volumes" Oct 31 00:43:45.253799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:45.253761 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:46.040366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:46.040329 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.198511 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:48.198465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.351466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:48.351431 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.591123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:48.591090 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:50.283207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:50.283157 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:50.780214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:50.780147 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:50.945261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:50.945222 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:51.125705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:51.125610 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:51.296840 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:51.296801 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:51.834297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:51.834257 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:54.899403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:54.899353 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:55.147415 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:55.147384 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.184500 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:56.184465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.476347 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:43:57.476314 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:43:59.323962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:59.323926 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:59.521782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:43:59.521742 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.200638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:00.200593 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:44:00.946446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:00.946405 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt"] Oct 31 00:44:00.946851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:00.946644 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" podUID="e408554a-a619-44f0-ace6-afe9c9188a31" containerName="discovery" containerID="cri-o://b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1" gracePeriod=30 Oct 31 00:44:01.212885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.212823 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:44:01.294473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294439 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.294473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294474 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.294737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294528 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.294737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294580 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.294737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294644 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.294965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294745 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.294965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.294841 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig\") pod \"e408554a-a619-44f0-ace6-afe9c9188a31\" (UID: \"e408554a-a619-44f0-ace6-afe9c9188a31\") " Oct 31 00:44:01.295465 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.295374 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:01.297365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.297336 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:01.297365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.297352 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:01.297971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.297947 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:01.298107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.297990 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:01.298107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.298038 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:01.298271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.298221 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x" pod "e408554a-a619-44f0-ace6-afe9c9188a31" (UID: "e408554a-a619-44f0-ace6-afe9c9188a31"). InnerVolumeSpecName "kube-api-access-64d7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:01.356638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.356596 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.396352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396319 5943 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-cacerts\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.396352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396348 5943 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e408554a-a619-44f0-ace6-afe9c9188a31-local-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.396352 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396359 5943 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-dns-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.396571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396370 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64d7x\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-kube-api-access-64d7x\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.396571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396380 5943 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e408554a-a619-44f0-ace6-afe9c9188a31-istio-csr-ca-configmap\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.396571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396388 5943 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e408554a-a619-44f0-ace6-afe9c9188a31-istio-kubeconfig\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.396571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:01.396396 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e408554a-a619-44f0-ace6-afe9c9188a31-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:02.010924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.010877 5943 generic.go:356] "Generic (PLEG): container finished" podID="e408554a-a619-44f0-ace6-afe9c9188a31" containerID="b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1" exitCode=0 Oct 31 00:44:02.011385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.010931 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" Oct 31 00:44:02.011385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.010971 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" event={"ID":"e408554a-a619-44f0-ace6-afe9c9188a31","Type":"ContainerDied","Data":"b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1"} Oct 31 00:44:02.011385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.011036 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt" event={"ID":"e408554a-a619-44f0-ace6-afe9c9188a31","Type":"ContainerDied","Data":"0a0911478f565f0fcb82df91309879960b5769672a589f2ddd5ea455a78ce767"} Oct 31 00:44:02.011385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.011064 5943 scope.go:122] "RemoveContainer" containerID="b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1" Oct 31 00:44:02.022812 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.022786 5943 scope.go:122] "RemoveContainer" containerID="b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1" Oct 31 00:44:02.023138 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:44:02.023121 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1\": container with ID starting with b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1 not found: ID does not exist" containerID="b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1" Oct 31 00:44:02.023209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.023147 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1"} err="failed to get container status \"b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1\": rpc error: code = NotFound desc = could not find container \"b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1\": container with ID starting with b5ded45ff76d6d5a0e357b933a09c2e3015b42b2a1842b7d2d9ed627b574eca1 not found: ID does not exist" Oct 31 00:44:02.177314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.177268 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt"] Oct 31 00:44:02.226964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:02.226921 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lvhqt"] Oct 31 00:44:04.041354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:04.041318 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e408554a-a619-44f0-ace6-afe9c9188a31" path="/var/lib/kubelet/pods/e408554a-a619-44f0-ace6-afe9c9188a31/volumes" Oct 31 00:44:06.186022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:06.185989 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:11.002129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:11.002078 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:15.736562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:15.736530 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:16.100697 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:16.100654 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz"] Oct 31 00:44:16.101248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:16.100949 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" containerID="cri-o://58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9" gracePeriod=30 Oct 31 00:44:16.178659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:16.178628 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:44:16.178833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:16.178703 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:20.479925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:20.479847 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:20.636430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:20.636392 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:20.783123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:20.783082 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:21.370164 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.370099 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:44:21.434321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434289 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434327 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434349 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434377 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434460 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434492 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434536 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434627 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434628 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:21.434869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434643 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:21.434869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434711 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs\") pod \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\" (UID: \"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7\") " Oct 31 00:44:21.434869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434854 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:21.435125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.434876 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:21.435125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.435099 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:21.435556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.435530 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.435556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.435558 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.435756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.435573 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.435756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.435586 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.435756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.435600 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.436903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.436875 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:21.436999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.436964 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:21.436999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.436965 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:21.436999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.436990 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb" pod "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" (UID: "ea0c55fa-7995-49b8-9fd0-81c664d7b8a7"). InnerVolumeSpecName "kube-api-access-6n6pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:21.536980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.536941 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.536980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.536972 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.536980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.536986 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:21.537447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:21.536999 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6pb\" (UniqueName: \"kubernetes.io/projected/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7-kube-api-access-6n6pb\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.140446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.140410 5943 generic.go:356] "Generic (PLEG): container finished" podID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerID="58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9" exitCode=0 Oct 31 00:44:22.140651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.140456 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" event={"ID":"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7","Type":"ContainerDied","Data":"58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9"} Oct 31 00:44:22.140651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.140463 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" Oct 31 00:44:22.140651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.140504 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz" event={"ID":"ea0c55fa-7995-49b8-9fd0-81c664d7b8a7","Type":"ContainerDied","Data":"61c1c4e4f2fe341b9904fcfa6cf299ad873a0a4e33189f394d11b006f9c38d77"} Oct 31 00:44:22.140651 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.140526 5943 scope.go:122] "RemoveContainer" containerID="58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9" Oct 31 00:44:22.151089 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.151069 5943 scope.go:122] "RemoveContainer" containerID="58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9" Oct 31 00:44:22.151419 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:44:22.151390 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9\": container with ID starting with 58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9 not found: ID does not exist" containerID="58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9" Oct 31 00:44:22.151516 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.151433 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9"} err="failed to get container status \"58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9\": rpc error: code = NotFound desc = could not find container \"58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9\": container with ID starting with 58e2342093d8c6ae1c142091dce623c9ee75292a7352ff5be2b01dca63f5fea9 not found: ID does not exist" Oct 31 00:44:22.354645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.354552 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz"] Oct 31 00:44:22.404144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:22.404109 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dbxkz"] Oct 31 00:44:24.041452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:24.041418 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" path="/var/lib/kubelet/pods/ea0c55fa-7995-49b8-9fd0-81c664d7b8a7/volumes" Oct 31 00:44:25.434734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:25.434699 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.477318 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:44:27.477277 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:44:30.297084 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:30.297042 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:35.058941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:35.058908 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:39.790699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:39.790661 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.802698 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:40.802662 5943 scope.go:122] "RemoveContainer" containerID="f6823099e8612c19d69c6c8587be0d928c984f963a3443c3102b2ef0386a0368" Oct 31 00:44:41.186257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:41.186158 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:44:44.516069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:44.516023 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:44.692676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:44.692633 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:44.850272 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:44.850162 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:46.911229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:46.911165 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.143092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.143051 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.695141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695103 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh"] Oct 31 00:44:48.695647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695633 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" Oct 31 00:44:48.695693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695649 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" Oct 31 00:44:48.695693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695672 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" containerName="revoke-keys" Oct 31 00:44:48.695693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695681 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" containerName="revoke-keys" Oct 31 00:44:48.695790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695698 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e408554a-a619-44f0-ace6-afe9c9188a31" containerName="discovery" Oct 31 00:44:48.695790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695704 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e408554a-a619-44f0-ace6-afe9c9188a31" containerName="discovery" Oct 31 00:44:48.695790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695788 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0c55fa-7995-49b8-9fd0-81c664d7b8a7" containerName="istio-proxy" Oct 31 00:44:48.695878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695801 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a68b40-0ff9-4359-83dd-2b5b04a8fbed" containerName="revoke-keys" Oct 31 00:44:48.695878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.695810 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e408554a-a619-44f0-ace6-afe9c9188a31" containerName="discovery" Oct 31 00:44:48.700802 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.700784 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.758022 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ef8827-openshift-default-dockercfg-gqgnl\"" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759311 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759349 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759430 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759469 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759494 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759571 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759618 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxtz\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759669 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.759891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.759696 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.791888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.791852 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.809204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.809157 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh"] Oct 31 00:44:48.860705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860667 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.860937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860717 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.860937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860749 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.860937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860857 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.860937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860913 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxtz\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860952 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.860980 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.861015 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.861040 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.861170 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.861384 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.861713 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.861881 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.861808 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.862100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.862077 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.863601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.863578 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.863709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.863613 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.944505 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.944475 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxtz\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:48.954424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:48.954352 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token\") pod \"e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:49.016909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.016861 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:49.112749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.112718 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:49.460429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.460401 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh"] Oct 31 00:44:49.462986 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:44:49.462961 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652b4620_ca12_4f36_910c_057905987c2e.slice/crio-52b6655555e332dedeb056c493bb495ae2b262e08f9c7dfdda32ac352f79fefc WatchSource:0}: Error finding container 52b6655555e332dedeb056c493bb495ae2b262e08f9c7dfdda32ac352f79fefc: Status 404 returned error can't find the container with id 52b6655555e332dedeb056c493bb495ae2b262e08f9c7dfdda32ac352f79fefc Oct 31 00:44:49.465983 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.465947 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:44:49.466102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.466028 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:44:49.466102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.466078 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:44:49.965853 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:49.965814 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:50.313538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:50.313501 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" event={"ID":"652b4620-ca12-4f36-910c-057905987c2e","Type":"ContainerStarted","Data":"aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d"} Oct 31 00:44:50.313748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:50.313568 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" event={"ID":"652b4620-ca12-4f36-910c-057905987c2e","Type":"ContainerStarted","Data":"52b6655555e332dedeb056c493bb495ae2b262e08f9c7dfdda32ac352f79fefc"} Oct 31 00:44:50.483968 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:50.483906 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podStartSLOduration=2.483889252 podStartE2EDuration="2.483889252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:50.483138889 +0000 UTC m=+2470.576429099" watchObservedRunningTime="2026-10-31 00:44:50.483889252 +0000 UTC m=+2470.577179441" Oct 31 00:44:51.017071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:51.017041 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:51.023614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:51.023589 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:51.046309 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:51.046279 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p"] Oct 31 00:44:51.049001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:51.048979 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef70ff2f-qp47p"] Oct 31 00:44:51.320228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:51.320119 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:51.321097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:51.321073 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:44:52.041000 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:52.040964 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72688853-91e7-49ac-8133-da1bd472ade8" path="/var/lib/kubelet/pods/72688853-91e7-49ac-8133-da1bd472ade8/volumes" Oct 31 00:44:52.187689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:52.187656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:53.375012 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:53.374978 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:53.551338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:53.551302 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:53.696585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:53.696498 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:55.024023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:55.023985 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:57.476465 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:44:57.476424 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:44:59.170536 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.170496 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:59.368687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.368650 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:59.526511 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.526470 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:59.869476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.869385 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm"] Oct 31 00:44:59.873668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.873647 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:44:59.976145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.976110 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ef8827-dockercfg-pjd86\"" Oct 31 00:44:59.976145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.976139 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ef8827\"" Oct 31 00:44:59.978288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.978268 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm"] Oct 31 00:44:59.987704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.987676 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:44:59.987845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:44:59.987750 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92rwv\" (UniqueName: \"kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv\") pod \"payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:00.089080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.089047 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:00.089272 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.089125 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92rwv\" (UniqueName: \"kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv\") pod \"payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:00.089829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.089810 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:00.177024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.176948 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92rwv\" (UniqueName: \"kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv\") pod \"payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:00.184170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.184141 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n"] Oct 31 00:45:00.188062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.188036 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:00.188345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.188324 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7"] Oct 31 00:45:00.188463 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.188445 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.192051 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.192030 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj"] Oct 31 00:45:00.192166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.192119 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.195953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.195934 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v"] Oct 31 00:45:00.196041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.196024 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.200041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.200019 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.238601 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.238573 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7"] Oct 31 00:45:00.243724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.243264 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.244073 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.244027 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw"] Oct 31 00:45:00.248443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.248405 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:45:00.248736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.248718 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.254860 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.254788 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:45:00.256094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.256069 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:45:00.263124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.263092 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n"] Oct 31 00:45:00.284106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.283622 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8"] Oct 31 00:45:00.289679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.289643 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.290355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290329 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7"] Oct 31 00:45:00.290746 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290715 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7t4\" (UniqueName: \"kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4\") pod \"maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.290836 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290816 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-x5ttj\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.290897 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290849 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.290944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290925 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.290991 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290971 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.291034 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.290997 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4bx\" (UniqueName: \"kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.291092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.291052 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.291137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.291113 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99hf\" (UniqueName: \"kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.291170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.291136 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bbzf\" (UniqueName: \"kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf\") pod \"maas-api-key-cleanup-29890125-x5ttj\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.291245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.291169 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.291245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.291233 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfb6\" (UniqueName: \"kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6\") pod \"maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.291319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.291295 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.315687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.313943 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v"] Oct 31 00:45:00.318576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.318547 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj"] Oct 31 00:45:00.322006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.321769 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7"] Oct 31 00:45:00.340938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.339642 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8"] Oct 31 00:45:00.386293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.386258 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw"] Oct 31 00:45:00.391990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.391961 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfb6\" (UniqueName: \"kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6\") pod \"maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.392246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392025 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.392246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392087 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.392246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392108 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7t4\" (UniqueName: \"kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4\") pod \"maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.392246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392223 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-x5ttj\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.392504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392251 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.392504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392283 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.392504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392314 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8q7\" (UniqueName: \"kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7\") pod \"maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.392504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392354 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.392504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392390 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4bx\" (UniqueName: \"kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.392504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392474 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.392831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392518 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99hf\" (UniqueName: \"kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.392831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392554 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bbzf\" (UniqueName: \"kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf\") pod \"maas-api-key-cleanup-29890125-x5ttj\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.392831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.392610 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.393084 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.393051 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-x5ttj\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.393148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.393100 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.393315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.393297 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.393402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.393379 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.393893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.393873 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.394244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.394223 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.454267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.454162 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfb6\" (UniqueName: \"kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6\") pod \"maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.454267 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.454233 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7t4\" (UniqueName: \"kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4\") pod \"maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.454440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.454416 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.455218 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.455196 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4bx\" (UniqueName: \"kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.455467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.455445 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bbzf\" (UniqueName: \"kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf\") pod \"maas-api-key-cleanup-29890125-x5ttj\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.459159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.459134 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99hf\" (UniqueName: \"kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.493902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.493871 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8q7\" (UniqueName: \"kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7\") pod \"maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.494059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.494044 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.494665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.494647 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.504658 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.504635 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:00.534194 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.534150 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:00.542957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.542928 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:00.551215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.551161 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:00.563133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.563107 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8q7\" (UniqueName: \"kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7\") pod \"maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.582809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.582774 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm"] Oct 31 00:45:00.589306 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.589250 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:00.594139 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.594104 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:00.605944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.605888 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:00.729500 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.729446 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.886364 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.886341 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n"] Oct 31 00:45:00.888981 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:00.888940 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8702a95d_da73_43c0_8243_2e0991718513.slice/crio-adce5c90a35cad5de0382ff7b90cfe8f731af5393f8e7535db8d0413cf641c17 WatchSource:0}: Error finding container adce5c90a35cad5de0382ff7b90cfe8f731af5393f8e7535db8d0413cf641c17: Status 404 returned error can't find the container with id adce5c90a35cad5de0382ff7b90cfe8f731af5393f8e7535db8d0413cf641c17 Oct 31 00:45:00.953001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.952976 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7"] Oct 31 00:45:00.955259 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:00.955231 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45febe8_e6e8_4230_b06c_5791e67b5ac6.slice/crio-605acc38f314d93da92c86da2eba0a22c9fd98de6a65216fbb6d32f5d74a7f31 WatchSource:0}: Error finding container 605acc38f314d93da92c86da2eba0a22c9fd98de6a65216fbb6d32f5d74a7f31: Status 404 returned error can't find the container with id 605acc38f314d93da92c86da2eba0a22c9fd98de6a65216fbb6d32f5d74a7f31 Oct 31 00:45:00.976403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.976376 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v"] Oct 31 00:45:00.977062 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:00.977039 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cb92f6_92cd_4265_8605_d609fd387060.slice/crio-e44cd947eb06904986b6e6cc301c03b0aab85efafbce93ce49cd1d81815bf31a WatchSource:0}: Error finding container e44cd947eb06904986b6e6cc301c03b0aab85efafbce93ce49cd1d81815bf31a: Status 404 returned error can't find the container with id e44cd947eb06904986b6e6cc301c03b0aab85efafbce93ce49cd1d81815bf31a Oct 31 00:45:00.997696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:00.997668 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj"] Oct 31 00:45:00.998573 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:00.998545 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153a9803_ee8e_41f3_92fd_6e62b33a4d87.slice/crio-5526b50e87ae33b5fa07fdf0c6712dd7a7589300cff75ee94c247248f0018c4c WatchSource:0}: Error finding container 5526b50e87ae33b5fa07fdf0c6712dd7a7589300cff75ee94c247248f0018c4c: Status 404 returned error can't find the container with id 5526b50e87ae33b5fa07fdf0c6712dd7a7589300cff75ee94c247248f0018c4c Oct 31 00:45:01.037521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.037493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:01.126993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.126946 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw"] Oct 31 00:45:01.134238 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.134206 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8"] Oct 31 00:45:01.135557 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:01.135534 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6657ab_6e3e_4bbf_b2cf_dc322597b7e8.slice/crio-064cf141329ebab7fdf424bd704c2a82889eb22b7fececc30e8fd29941a7b47e WatchSource:0}: Error finding container 064cf141329ebab7fdf424bd704c2a82889eb22b7fececc30e8fd29941a7b47e: Status 404 returned error can't find the container with id 064cf141329ebab7fdf424bd704c2a82889eb22b7fececc30e8fd29941a7b47e Oct 31 00:45:01.136392 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:01.136371 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0759aa09_4f8b_48f1_b005_ae52866a6e85.slice/crio-d4eae0c71919bc8d2ba517ed5462c65b69c9a1566b4c2c12eed69f5935035dfe WatchSource:0}: Error finding container d4eae0c71919bc8d2ba517ed5462c65b69c9a1566b4c2c12eed69f5935035dfe: Status 404 returned error can't find the container with id d4eae0c71919bc8d2ba517ed5462c65b69c9a1566b4c2c12eed69f5935035dfe Oct 31 00:45:01.209770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.209524 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7"] Oct 31 00:45:01.281427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.281337 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:01.325239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.325208 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h"] Oct 31 00:45:01.328592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.328565 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-726d5b-zhx4h"] Oct 31 00:45:01.331490 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:01.331450 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fe4f43_a807_458e_a5df_7acee6e72e0a.slice/crio-c0d829b6151d08c0ee517de9dc6bec0b831b3df0048da007f105a6fd9e69255f WatchSource:0}: Error finding container c0d829b6151d08c0ee517de9dc6bec0b831b3df0048da007f105a6fd9e69255f: Status 404 returned error can't find the container with id c0d829b6151d08c0ee517de9dc6bec0b831b3df0048da007f105a6fd9e69255f Oct 31 00:45:01.387120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.387085 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerStarted","Data":"064cf141329ebab7fdf424bd704c2a82889eb22b7fececc30e8fd29941a7b47e"} Oct 31 00:45:01.388552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.388523 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerStarted","Data":"d4eae0c71919bc8d2ba517ed5462c65b69c9a1566b4c2c12eed69f5935035dfe"} Oct 31 00:45:01.390025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.389998 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerStarted","Data":"c0d829b6151d08c0ee517de9dc6bec0b831b3df0048da007f105a6fd9e69255f"} Oct 31 00:45:01.391789 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.391519 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerStarted","Data":"e44cd947eb06904986b6e6cc301c03b0aab85efafbce93ce49cd1d81815bf31a"} Oct 31 00:45:01.393087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.393058 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerStarted","Data":"5526b50e87ae33b5fa07fdf0c6712dd7a7589300cff75ee94c247248f0018c4c"} Oct 31 00:45:01.394934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.394906 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerStarted","Data":"df79bdb0290c3e4239ab533aa658c60a121d4aeec4192928c994aad69610265c"} Oct 31 00:45:01.395042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.394964 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerStarted","Data":"605acc38f314d93da92c86da2eba0a22c9fd98de6a65216fbb6d32f5d74a7f31"} Oct 31 00:45:01.397695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.397310 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerStarted","Data":"d486cfab6e8603397c9466fae3646a336a588814f1542b01426479c3c039c29e"} Oct 31 00:45:01.397695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.397354 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerStarted","Data":"adce5c90a35cad5de0382ff7b90cfe8f731af5393f8e7535db8d0413cf641c17"} Oct 31 00:45:01.399737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.399716 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" event={"ID":"84e9b1db-5573-41cd-a69f-035dd1854a73","Type":"ContainerStarted","Data":"bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1"} Oct 31 00:45:01.399823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.399751 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" event={"ID":"84e9b1db-5573-41cd-a69f-035dd1854a73","Type":"ContainerStarted","Data":"b2e7c0fdc2b4dde94d3efc0bf5f40c3e3211150756832ab0962fd5132f4c304d"} Oct 31 00:45:01.399823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.399793 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:01.511917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.511864 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" podStartSLOduration=1.511851386 podStartE2EDuration="1.511851386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.509527108 +0000 UTC m=+2481.602817332" watchObservedRunningTime="2026-10-31 00:45:01.511851386 +0000 UTC m=+2481.605141573" Oct 31 00:45:01.632531 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.632467 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" podStartSLOduration=1.632446332 podStartE2EDuration="1.632446332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.629859725 +0000 UTC m=+2481.723149940" watchObservedRunningTime="2026-10-31 00:45:01.632446332 +0000 UTC m=+2481.725736521" Oct 31 00:45:01.725709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.725667 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:01.746914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:01.746851 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" podStartSLOduration=2.746830212 podStartE2EDuration="2.746830212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.744572398 +0000 UTC m=+2481.837862628" watchObservedRunningTime="2026-10-31 00:45:01.746830212 +0000 UTC m=+2481.840120401" Oct 31 00:45:02.044686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.044653 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b82f48-1f68-4c8d-9688-da19d3b3e9fb" path="/var/lib/kubelet/pods/c1b82f48-1f68-4c8d-9688-da19d3b3e9fb/volumes" Oct 31 00:45:02.409146 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.409115 5943 generic.go:356] "Generic (PLEG): container finished" podID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerID="590394a355f8efcdd74c7a7944c713f3be5bd576ee4cdff64f973b63779bd809" exitCode=60 Oct 31 00:45:02.409568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.409206 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerDied","Data":"590394a355f8efcdd74c7a7944c713f3be5bd576ee4cdff64f973b63779bd809"} Oct 31 00:45:02.409568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.409487 5943 scope.go:122] "RemoveContainer" containerID="590394a355f8efcdd74c7a7944c713f3be5bd576ee4cdff64f973b63779bd809" Oct 31 00:45:02.411087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.411067 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerStarted","Data":"b6914caacb7e0a9cc8a897ec71d374f787e901f62e16e2e0a0870c30bfbbd125"} Oct 31 00:45:02.412753 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.412732 5943 generic.go:356] "Generic (PLEG): container finished" podID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerID="1430d6b2cb222089c0a57c3b7318a2edf9bcc6024d4b67c94ba570dac5a4ee5c" exitCode=60 Oct 31 00:45:02.412816 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.412793 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerDied","Data":"1430d6b2cb222089c0a57c3b7318a2edf9bcc6024d4b67c94ba570dac5a4ee5c"} Oct 31 00:45:02.413002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.412989 5943 scope.go:122] "RemoveContainer" containerID="1430d6b2cb222089c0a57c3b7318a2edf9bcc6024d4b67c94ba570dac5a4ee5c" Oct 31 00:45:02.415331 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.415311 5943 generic.go:356] "Generic (PLEG): container finished" podID="63cb92f6-92cd-4265-8605-d609fd387060" containerID="a1bceb4a943fa99e1f5a41cb4a9931daf462d704a6de4475f5fdcbc4fa2d7b63" exitCode=60 Oct 31 00:45:02.415422 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.415398 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerDied","Data":"a1bceb4a943fa99e1f5a41cb4a9931daf462d704a6de4475f5fdcbc4fa2d7b63"} Oct 31 00:45:02.415588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.415573 5943 scope.go:122] "RemoveContainer" containerID="a1bceb4a943fa99e1f5a41cb4a9931daf462d704a6de4475f5fdcbc4fa2d7b63" Oct 31 00:45:02.417070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.417052 5943 generic.go:356] "Generic (PLEG): container finished" podID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerID="5cc943d5090fda275edd93d7979c9481d2c12d54e384ce71164d369399ec0f71" exitCode=60 Oct 31 00:45:02.417162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.417098 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerDied","Data":"5cc943d5090fda275edd93d7979c9481d2c12d54e384ce71164d369399ec0f71"} Oct 31 00:45:02.417367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.417352 5943 scope.go:122] "RemoveContainer" containerID="5cc943d5090fda275edd93d7979c9481d2c12d54e384ce71164d369399ec0f71" Oct 31 00:45:02.418797 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.418775 5943 generic.go:356] "Generic (PLEG): container finished" podID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerID="df79bdb0290c3e4239ab533aa658c60a121d4aeec4192928c994aad69610265c" exitCode=60 Oct 31 00:45:02.418900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.418837 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerDied","Data":"df79bdb0290c3e4239ab533aa658c60a121d4aeec4192928c994aad69610265c"} Oct 31 00:45:02.419017 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.419003 5943 scope.go:122] "RemoveContainer" containerID="df79bdb0290c3e4239ab533aa658c60a121d4aeec4192928c994aad69610265c" Oct 31 00:45:02.420888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.420865 5943 generic.go:356] "Generic (PLEG): container finished" podID="8702a95d-da73-43c0-8243-2e0991718513" containerID="d486cfab6e8603397c9466fae3646a336a588814f1542b01426479c3c039c29e" exitCode=60 Oct 31 00:45:02.420970 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.420956 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerDied","Data":"d486cfab6e8603397c9466fae3646a336a588814f1542b01426479c3c039c29e"} Oct 31 00:45:02.421319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.421227 5943 scope.go:122] "RemoveContainer" containerID="d486cfab6e8603397c9466fae3646a336a588814f1542b01426479c3c039c29e" Oct 31 00:45:02.648002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:02.647903 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" podStartSLOduration=2.647884266 podStartE2EDuration="2.647884266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.645494073 +0000 UTC m=+2482.738784286" watchObservedRunningTime="2026-10-31 00:45:02.647884266 +0000 UTC m=+2482.741174453" Oct 31 00:45:03.430954 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.430850 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerStarted","Data":"8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69"} Oct 31 00:45:03.435277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.433600 5943 generic.go:356] "Generic (PLEG): container finished" podID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerID="b6914caacb7e0a9cc8a897ec71d374f787e901f62e16e2e0a0870c30bfbbd125" exitCode=60 Oct 31 00:45:03.435277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.433703 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerDied","Data":"b6914caacb7e0a9cc8a897ec71d374f787e901f62e16e2e0a0870c30bfbbd125"} Oct 31 00:45:03.435277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.434083 5943 scope.go:122] "RemoveContainer" containerID="b6914caacb7e0a9cc8a897ec71d374f787e901f62e16e2e0a0870c30bfbbd125" Oct 31 00:45:03.439423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.438225 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerStarted","Data":"9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b"} Oct 31 00:45:03.440696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.440634 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerStarted","Data":"6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6"} Oct 31 00:45:03.442980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.442954 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerStarted","Data":"f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281"} Oct 31 00:45:03.445303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.445037 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerStarted","Data":"959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126"} Oct 31 00:45:03.447336 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.447316 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerStarted","Data":"3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b"} Oct 31 00:45:03.447869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.447629 5943 scope.go:122] "RemoveContainer" containerID="3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b" Oct 31 00:45:03.447869 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:03.447828 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n_odh-ai-gateway-infra(8702a95d-da73-43c0-8243-2e0991718513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" podUID="8702a95d-da73-43c0-8243-2e0991718513" Oct 31 00:45:03.547715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.547665 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" podStartSLOduration=3.547649537 podStartE2EDuration="3.547649537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.546087853 +0000 UTC m=+2483.639378074" watchObservedRunningTime="2026-10-31 00:45:03.547649537 +0000 UTC m=+2483.640939724" Oct 31 00:45:03.898687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:03.898630 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" podStartSLOduration=3.898616589 podStartE2EDuration="3.898616589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.89801539 +0000 UTC m=+2483.991305597" watchObservedRunningTime="2026-10-31 00:45:03.898616589 +0000 UTC m=+2483.991906815" Oct 31 00:45:04.132105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.132051 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" podStartSLOduration=4.13203268 podStartE2EDuration="4.13203268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.130210167 +0000 UTC m=+2484.223500354" watchObservedRunningTime="2026-10-31 00:45:04.13203268 +0000 UTC m=+2484.225322868" Oct 31 00:45:04.244900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.244848 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" podStartSLOduration=4.244833592 podStartE2EDuration="4.244833592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.244075092 +0000 UTC m=+2484.337365299" watchObservedRunningTime="2026-10-31 00:45:04.244833592 +0000 UTC m=+2484.338123778" Oct 31 00:45:04.456279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.456235 5943 generic.go:356] "Generic (PLEG): container finished" podID="63cb92f6-92cd-4265-8605-d609fd387060" containerID="6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6" exitCode=60 Oct 31 00:45:04.456712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.456316 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerDied","Data":"6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6"} Oct 31 00:45:04.456712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.456377 5943 scope.go:122] "RemoveContainer" containerID="a1bceb4a943fa99e1f5a41cb4a9931daf462d704a6de4475f5fdcbc4fa2d7b63" Oct 31 00:45:04.456712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.456576 5943 scope.go:122] "RemoveContainer" containerID="6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6" Oct 31 00:45:04.456901 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.456878 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v_odh-ai-gateway-infra(63cb92f6-92cd-4265-8605-d609fd387060)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" podUID="63cb92f6-92cd-4265-8605-d609fd387060" Oct 31 00:45:04.458615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.458593 5943 generic.go:356] "Generic (PLEG): container finished" podID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerID="f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281" exitCode=60 Oct 31 00:45:04.458716 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.458669 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerDied","Data":"f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281"} Oct 31 00:45:04.458960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.458936 5943 scope.go:122] "RemoveContainer" containerID="f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281" Oct 31 00:45:04.459287 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.459266 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-x5ttj_odh-ai-gateway-infra(153a9803-ee8e-41f3-92fd-6e62b33a4d87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" Oct 31 00:45:04.460773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.460753 5943 generic.go:356] "Generic (PLEG): container finished" podID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerID="959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126" exitCode=60 Oct 31 00:45:04.460886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.460831 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerDied","Data":"959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126"} Oct 31 00:45:04.461064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.461050 5943 scope.go:122] "RemoveContainer" containerID="959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126" Oct 31 00:45:04.461362 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.461338 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7_odh-ai-gateway-infra(b45febe8-e6e8-4230-b06c-5791e67b5ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" Oct 31 00:45:04.462857 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.462837 5943 generic.go:356] "Generic (PLEG): container finished" podID="8702a95d-da73-43c0-8243-2e0991718513" containerID="3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b" exitCode=60 Oct 31 00:45:04.462940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.462917 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerDied","Data":"3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b"} Oct 31 00:45:04.463281 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.463262 5943 scope.go:122] "RemoveContainer" containerID="3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b" Oct 31 00:45:04.463575 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.463544 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n_odh-ai-gateway-infra(8702a95d-da73-43c0-8243-2e0991718513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" podUID="8702a95d-da73-43c0-8243-2e0991718513" Oct 31 00:45:04.465007 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.464986 5943 generic.go:356] "Generic (PLEG): container finished" podID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerID="8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69" exitCode=60 Oct 31 00:45:04.465103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.465063 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerDied","Data":"8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69"} Oct 31 00:45:04.465315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.465297 5943 scope.go:122] "RemoveContainer" containerID="8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69" Oct 31 00:45:04.465577 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.465555 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8_odh-ai-gateway-infra(5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" Oct 31 00:45:04.466960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.466938 5943 generic.go:356] "Generic (PLEG): container finished" podID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerID="dc6a736c82d609196af990768cf09ee1b4dc315699be6cc674c55b1d386bebd4" exitCode=60 Oct 31 00:45:04.467046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.467010 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerDied","Data":"dc6a736c82d609196af990768cf09ee1b4dc315699be6cc674c55b1d386bebd4"} Oct 31 00:45:04.467310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.467290 5943 scope.go:122] "RemoveContainer" containerID="dc6a736c82d609196af990768cf09ee1b4dc315699be6cc674c55b1d386bebd4" Oct 31 00:45:04.467556 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.467536 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw_odh-ai-gateway-infra(0759aa09-4f8b-48f1-b005-ae52866a6e85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" Oct 31 00:45:04.469003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.468987 5943 generic.go:356] "Generic (PLEG): container finished" podID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerID="9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b" exitCode=60 Oct 31 00:45:04.469083 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.469038 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerDied","Data":"9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b"} Oct 31 00:45:04.469311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.469296 5943 scope.go:122] "RemoveContainer" containerID="9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b" Oct 31 00:45:04.469519 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:04.469503 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7_odh-ai-gateway-infra(10fe4f43-a807-458e-a5df-7acee6e72e0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" Oct 31 00:45:04.470129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.470108 5943 scope.go:122] "RemoveContainer" containerID="5cc943d5090fda275edd93d7979c9481d2c12d54e384ce71164d369399ec0f71" Oct 31 00:45:04.482921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.482900 5943 scope.go:122] "RemoveContainer" containerID="df79bdb0290c3e4239ab533aa658c60a121d4aeec4192928c994aad69610265c" Oct 31 00:45:04.498262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.498235 5943 scope.go:122] "RemoveContainer" containerID="d486cfab6e8603397c9466fae3646a336a588814f1542b01426479c3c039c29e" Oct 31 00:45:04.513818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.513077 5943 scope.go:122] "RemoveContainer" containerID="590394a355f8efcdd74c7a7944c713f3be5bd576ee4cdff64f973b63779bd809" Oct 31 00:45:04.527553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.527536 5943 scope.go:122] "RemoveContainer" containerID="b6914caacb7e0a9cc8a897ec71d374f787e901f62e16e2e0a0870c30bfbbd125" Oct 31 00:45:04.540303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:04.540281 5943 scope.go:122] "RemoveContainer" containerID="1430d6b2cb222089c0a57c3b7318a2edf9bcc6024d4b67c94ba570dac5a4ee5c" Oct 31 00:45:05.477274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.477247 5943 scope.go:122] "RemoveContainer" containerID="8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69" Oct 31 00:45:05.477711 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.477574 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8_odh-ai-gateway-infra(5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" Oct 31 00:45:05.478892 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.478872 5943 scope.go:122] "RemoveContainer" containerID="dc6a736c82d609196af990768cf09ee1b4dc315699be6cc674c55b1d386bebd4" Oct 31 00:45:05.479151 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.479129 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw_odh-ai-gateway-infra(0759aa09-4f8b-48f1-b005-ae52866a6e85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" Oct 31 00:45:05.480414 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.480385 5943 scope.go:122] "RemoveContainer" containerID="9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b" Oct 31 00:45:05.480649 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.480625 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7_odh-ai-gateway-infra(10fe4f43-a807-458e-a5df-7acee6e72e0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" Oct 31 00:45:05.481981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.481965 5943 scope.go:122] "RemoveContainer" containerID="6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6" Oct 31 00:45:05.482169 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.482150 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v_odh-ai-gateway-infra(63cb92f6-92cd-4265-8605-d609fd387060)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" podUID="63cb92f6-92cd-4265-8605-d609fd387060" Oct 31 00:45:05.483585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.483569 5943 scope.go:122] "RemoveContainer" containerID="f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281" Oct 31 00:45:05.483780 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.483758 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-x5ttj_odh-ai-gateway-infra(153a9803-ee8e-41f3-92fd-6e62b33a4d87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" Oct 31 00:45:05.485241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.485222 5943 scope.go:122] "RemoveContainer" containerID="959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126" Oct 31 00:45:05.485446 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.485425 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7_odh-ai-gateway-infra(b45febe8-e6e8-4230-b06c-5791e67b5ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" Oct 31 00:45:05.486723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:05.486708 5943 scope.go:122] "RemoveContainer" containerID="3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b" Oct 31 00:45:05.486873 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:05.486857 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n_odh-ai-gateway-infra(8702a95d-da73-43c0-8243-2e0991718513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" podUID="8702a95d-da73-43c0-8243-2e0991718513" Oct 31 00:45:06.288023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:06.287993 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:08.085334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:08.085303 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:08.318477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:08.318439 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:08.476202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:08.476092 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:11.158065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:11.158030 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:12.424401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:12.424373 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:45:13.441696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:13.441663 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:15.034468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:15.034436 5943 scope.go:122] "RemoveContainer" containerID="959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126" Oct 31 00:45:15.550116 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:15.550076 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerStarted","Data":"3578ec08c8807d3adaf09eb381508421a193bb5763f1f320f5568936741c1049"} Oct 31 00:45:15.967942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:15.967902 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:16.034433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.034401 5943 scope.go:122] "RemoveContainer" containerID="3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b" Oct 31 00:45:16.034619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.034522 5943 scope.go:122] "RemoveContainer" containerID="8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69" Oct 31 00:45:16.559498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.559464 5943 generic.go:356] "Generic (PLEG): container finished" podID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerID="3578ec08c8807d3adaf09eb381508421a193bb5763f1f320f5568936741c1049" exitCode=60 Oct 31 00:45:16.559690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.559540 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerDied","Data":"3578ec08c8807d3adaf09eb381508421a193bb5763f1f320f5568936741c1049"} Oct 31 00:45:16.559690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.559603 5943 scope.go:122] "RemoveContainer" containerID="959e32561bb0c59613fdae9c572869f640fdccc6673105b4413133fef8361126" Oct 31 00:45:16.560029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.560011 5943 scope.go:122] "RemoveContainer" containerID="3578ec08c8807d3adaf09eb381508421a193bb5763f1f320f5568936741c1049" Oct 31 00:45:16.560420 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:16.560380 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7_odh-ai-gateway-infra(b45febe8-e6e8-4230-b06c-5791e67b5ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" Oct 31 00:45:16.561862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.561835 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerStarted","Data":"b878c4414e8cb2750b8eb29f55ee8aa4599a060b2e9bdaa771141900ec751d68"} Oct 31 00:45:16.563735 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.563713 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerStarted","Data":"61af4d7c023a59fbec5499e0e124160918b826e22bddd85675fd32049c277fc7"} Oct 31 00:45:16.674927 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:16.674890 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7"] Oct 31 00:45:17.035094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.035060 5943 scope.go:122] "RemoveContainer" containerID="9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b" Oct 31 00:45:17.035579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.035269 5943 scope.go:122] "RemoveContainer" containerID="f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281" Oct 31 00:45:17.035579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.035392 5943 scope.go:122] "RemoveContainer" containerID="dc6a736c82d609196af990768cf09ee1b4dc315699be6cc674c55b1d386bebd4" Oct 31 00:45:17.035579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.035521 5943 scope.go:122] "RemoveContainer" containerID="6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6" Oct 31 00:45:17.575777 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.575745 5943 generic.go:356] "Generic (PLEG): container finished" podID="8702a95d-da73-43c0-8243-2e0991718513" containerID="b878c4414e8cb2750b8eb29f55ee8aa4599a060b2e9bdaa771141900ec751d68" exitCode=60 Oct 31 00:45:17.575900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.575819 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerDied","Data":"b878c4414e8cb2750b8eb29f55ee8aa4599a060b2e9bdaa771141900ec751d68"} Oct 31 00:45:17.575971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.575900 5943 scope.go:122] "RemoveContainer" containerID="3b3c5c7251e4cf40dc548ca39b6d033c30d22d2991e0001dc70fd5607351866b" Oct 31 00:45:17.576328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.576308 5943 scope.go:122] "RemoveContainer" containerID="b878c4414e8cb2750b8eb29f55ee8aa4599a060b2e9bdaa771141900ec751d68" Oct 31 00:45:17.576666 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:17.576643 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n_odh-ai-gateway-infra(8702a95d-da73-43c0-8243-2e0991718513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" podUID="8702a95d-da73-43c0-8243-2e0991718513" Oct 31 00:45:17.578257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.578232 5943 generic.go:356] "Generic (PLEG): container finished" podID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerID="61af4d7c023a59fbec5499e0e124160918b826e22bddd85675fd32049c277fc7" exitCode=60 Oct 31 00:45:17.578359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.578290 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerDied","Data":"61af4d7c023a59fbec5499e0e124160918b826e22bddd85675fd32049c277fc7"} Oct 31 00:45:17.578657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.578642 5943 scope.go:122] "RemoveContainer" containerID="61af4d7c023a59fbec5499e0e124160918b826e22bddd85675fd32049c277fc7" Oct 31 00:45:17.578932 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:17.578914 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8_odh-ai-gateway-infra(5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" Oct 31 00:45:17.580479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.580455 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerStarted","Data":"0e38749c1e353d0676ba725af5516a01be71581fcbc62fbdb44c8c1db70c40cc"} Oct 31 00:45:17.582615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.582590 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerStarted","Data":"96f22728aacc303a9c551cd113194114d31544547ee8c5e35388364e0223e893"} Oct 31 00:45:17.584734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.584703 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerStarted","Data":"253134bbee393748d4745650580ecd803fe8c17883d0dc97884ef8a39d9d7ffc"} Oct 31 00:45:17.586819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.586798 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerStarted","Data":"e59b06e2a951bf804c42e2d832a1ca78b9c327db935d46bce9a86163e6d47d66"} Oct 31 00:45:17.591071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.591055 5943 scope.go:122] "RemoveContainer" containerID="8755622de5dfea99c3ed6d0352bf7153bd5b6ccd6517f92d080da5283e8afc69" Oct 31 00:45:17.739758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.739723 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:17.812422 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.812384 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n"] Oct 31 00:45:17.813611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.813577 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44\") pod \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " Oct 31 00:45:17.813765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.813752 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca\") pod \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\" (UID: \"b45febe8-e6e8-4230-b06c-5791e67b5ac6\") " Oct 31 00:45:17.814124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.814099 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca" pod "b45febe8-e6e8-4230-b06c-5791e67b5ac6" (UID: "b45febe8-e6e8-4230-b06c-5791e67b5ac6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:17.815894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.815872 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44" pod "b45febe8-e6e8-4230-b06c-5791e67b5ac6" (UID: "b45febe8-e6e8-4230-b06c-5791e67b5ac6"). InnerVolumeSpecName "kube-api-access-mrh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:17.914912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.914826 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/b45febe8-e6e8-4230-b06c-5791e67b5ac6-kube-api-access-mrh44\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:17.914912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.914856 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b45febe8-e6e8-4230-b06c-5791e67b5ac6-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:17.937450 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:17.937413 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8"] Oct 31 00:45:18.593974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.593942 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" Oct 31 00:45:18.594529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.593940 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7" event={"ID":"b45febe8-e6e8-4230-b06c-5791e67b5ac6","Type":"ContainerDied","Data":"605acc38f314d93da92c86da2eba0a22c9fd98de6a65216fbb6d32f5d74a7f31"} Oct 31 00:45:18.594529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.594085 5943 scope.go:122] "RemoveContainer" containerID="3578ec08c8807d3adaf09eb381508421a193bb5763f1f320f5568936741c1049" Oct 31 00:45:18.601056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.601033 5943 generic.go:356] "Generic (PLEG): container finished" podID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerID="0e38749c1e353d0676ba725af5516a01be71581fcbc62fbdb44c8c1db70c40cc" exitCode=60 Oct 31 00:45:18.601167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.601108 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerDied","Data":"0e38749c1e353d0676ba725af5516a01be71581fcbc62fbdb44c8c1db70c40cc"} Oct 31 00:45:18.601568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.601550 5943 scope.go:122] "RemoveContainer" containerID="0e38749c1e353d0676ba725af5516a01be71581fcbc62fbdb44c8c1db70c40cc" Oct 31 00:45:18.601918 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:18.601897 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw_odh-ai-gateway-infra(0759aa09-4f8b-48f1-b005-ae52866a6e85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" Oct 31 00:45:18.603342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.603315 5943 generic.go:356] "Generic (PLEG): container finished" podID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerID="96f22728aacc303a9c551cd113194114d31544547ee8c5e35388364e0223e893" exitCode=60 Oct 31 00:45:18.603458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.603438 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerDied","Data":"96f22728aacc303a9c551cd113194114d31544547ee8c5e35388364e0223e893"} Oct 31 00:45:18.603823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.603805 5943 scope.go:122] "RemoveContainer" containerID="96f22728aacc303a9c551cd113194114d31544547ee8c5e35388364e0223e893" Oct 31 00:45:18.604118 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:18.604096 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7_odh-ai-gateway-infra(10fe4f43-a807-458e-a5df-7acee6e72e0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" Oct 31 00:45:18.605699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.605674 5943 generic.go:356] "Generic (PLEG): container finished" podID="63cb92f6-92cd-4265-8605-d609fd387060" containerID="253134bbee393748d4745650580ecd803fe8c17883d0dc97884ef8a39d9d7ffc" exitCode=60 Oct 31 00:45:18.605799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.605741 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerDied","Data":"253134bbee393748d4745650580ecd803fe8c17883d0dc97884ef8a39d9d7ffc"} Oct 31 00:45:18.605947 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.605932 5943 scope.go:122] "RemoveContainer" containerID="dc6a736c82d609196af990768cf09ee1b4dc315699be6cc674c55b1d386bebd4" Oct 31 00:45:18.606125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.606112 5943 scope.go:122] "RemoveContainer" containerID="253134bbee393748d4745650580ecd803fe8c17883d0dc97884ef8a39d9d7ffc" Oct 31 00:45:18.606444 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:18.606414 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v_odh-ai-gateway-infra(63cb92f6-92cd-4265-8605-d609fd387060)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" podUID="63cb92f6-92cd-4265-8605-d609fd387060" Oct 31 00:45:18.608578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.608554 5943 generic.go:356] "Generic (PLEG): container finished" podID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerID="e59b06e2a951bf804c42e2d832a1ca78b9c327db935d46bce9a86163e6d47d66" exitCode=60 Oct 31 00:45:18.608657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.608589 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerDied","Data":"e59b06e2a951bf804c42e2d832a1ca78b9c327db935d46bce9a86163e6d47d66"} Oct 31 00:45:18.608937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.608921 5943 scope.go:122] "RemoveContainer" containerID="e59b06e2a951bf804c42e2d832a1ca78b9c327db935d46bce9a86163e6d47d66" Oct 31 00:45:18.609273 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:18.609249 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-x5ttj_odh-ai-gateway-infra(153a9803-ee8e-41f3-92fd-6e62b33a4d87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" Oct 31 00:45:18.622642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.622615 5943 scope.go:122] "RemoveContainer" containerID="9738bfffee64b038ab258dd746ec3b5002e390dc3afc46225a75f484ba6f5f0b" Oct 31 00:45:18.647255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.646068 5943 scope.go:122] "RemoveContainer" containerID="6f6acad1d0fe99e0de8e4afab71456794c92fc27cde0966b96d26506a31345f6" Oct 31 00:45:18.660677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.660654 5943 scope.go:122] "RemoveContainer" containerID="f06f138fea59d24a17c61f4b53401062c011d68c87e75f77c144db8ede434281" Oct 31 00:45:18.777528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.777504 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:18.794982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.794869 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:18.812029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.812002 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj"] Oct 31 00:45:18.826522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.826493 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca\") pod \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " Oct 31 00:45:18.826690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.826581 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7\" (UniqueName: \"kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7\") pod \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\" (UID: \"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8\") " Oct 31 00:45:18.826690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.826627 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca\") pod \"8702a95d-da73-43c0-8243-2e0991718513\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " Oct 31 00:45:18.826690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.826670 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx\" (UniqueName: \"kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx\") pod \"8702a95d-da73-43c0-8243-2e0991718513\" (UID: \"8702a95d-da73-43c0-8243-2e0991718513\") " Oct 31 00:45:18.826969 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.826944 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca" pod "5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" (UID: "5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.827088 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.827058 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca" pod "8702a95d-da73-43c0-8243-2e0991718513" (UID: "8702a95d-da73-43c0-8243-2e0991718513"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.827248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.827233 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.827310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.827254 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8702a95d-da73-43c0-8243-2e0991718513-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.828881 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.828860 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx" pod "8702a95d-da73-43c0-8243-2e0991718513" (UID: "8702a95d-da73-43c0-8243-2e0991718513"). InnerVolumeSpecName "kube-api-access-zc4bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.828937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.828892 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7" pod "5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" (UID: "5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8"). InnerVolumeSpecName "kube-api-access-xj8q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.928315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.928226 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8q7\" (UniqueName: \"kubernetes.io/projected/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8-kube-api-access-xj8q7\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.928315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.928258 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc4bx\" (UniqueName: \"kubernetes.io/projected/8702a95d-da73-43c0-8243-2e0991718513-kube-api-access-zc4bx\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.928315 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.928282 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7"] Oct 31 00:45:18.992463 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.992427 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7"] Oct 31 00:45:18.994690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:18.994666 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890125-lw6f7"] Oct 31 00:45:19.156466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.156430 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v"] Oct 31 00:45:19.272210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.272160 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw"] Oct 31 00:45:19.632743 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.632717 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" Oct 31 00:45:19.633132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.632755 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n" event={"ID":"8702a95d-da73-43c0-8243-2e0991718513","Type":"ContainerDied","Data":"adce5c90a35cad5de0382ff7b90cfe8f731af5393f8e7535db8d0413cf641c17"} Oct 31 00:45:19.633132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.632837 5943 scope.go:122] "RemoveContainer" containerID="b878c4414e8cb2750b8eb29f55ee8aa4599a060b2e9bdaa771141900ec751d68" Oct 31 00:45:19.634867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.634804 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" Oct 31 00:45:19.634867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.634847 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8" event={"ID":"5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8","Type":"ContainerDied","Data":"064cf141329ebab7fdf424bd704c2a82889eb22b7fececc30e8fd29941a7b47e"} Oct 31 00:45:19.658484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.656340 5943 scope.go:122] "RemoveContainer" containerID="61af4d7c023a59fbec5499e0e124160918b826e22bddd85675fd32049c277fc7" Oct 31 00:45:19.872592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.872527 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:19.875826 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.875768 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:19.881404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.881381 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:19.884912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.884893 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:19.940762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.940727 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca\") pod \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " Oct 31 00:45:19.940762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.940767 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6\" (UniqueName: \"kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6\") pod \"0759aa09-4f8b-48f1-b005-ae52866a6e85\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " Oct 31 00:45:19.940999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.940812 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca\") pod \"10fe4f43-a807-458e-a5df-7acee6e72e0a\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " Oct 31 00:45:19.941065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.940968 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf\" (UniqueName: \"kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf\") pod \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\" (UID: \"153a9803-ee8e-41f3-92fd-6e62b33a4d87\") " Oct 31 00:45:19.941147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941129 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca\") pod \"0759aa09-4f8b-48f1-b005-ae52866a6e85\" (UID: \"0759aa09-4f8b-48f1-b005-ae52866a6e85\") " Oct 31 00:45:19.941235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941210 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca" pod "153a9803-ee8e-41f3-92fd-6e62b33a4d87" (UID: "153a9803-ee8e-41f3-92fd-6e62b33a4d87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.941320 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941253 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca" pod "10fe4f43-a807-458e-a5df-7acee6e72e0a" (UID: "10fe4f43-a807-458e-a5df-7acee6e72e0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.941320 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941312 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4\" (UniqueName: \"kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4\") pod \"10fe4f43-a807-458e-a5df-7acee6e72e0a\" (UID: \"10fe4f43-a807-458e-a5df-7acee6e72e0a\") " Oct 31 00:45:19.941494 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941473 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca\") pod \"63cb92f6-92cd-4265-8605-d609fd387060\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " Oct 31 00:45:19.941614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941517 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf\" (UniqueName: \"kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf\") pod \"63cb92f6-92cd-4265-8605-d609fd387060\" (UID: \"63cb92f6-92cd-4265-8605-d609fd387060\") " Oct 31 00:45:19.941691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.941517 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca" pod "0759aa09-4f8b-48f1-b005-ae52866a6e85" (UID: "0759aa09-4f8b-48f1-b005-ae52866a6e85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.942087 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.942050 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca" pod "63cb92f6-92cd-4265-8605-d609fd387060" (UID: "63cb92f6-92cd-4265-8605-d609fd387060"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.942242 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.942222 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0759aa09-4f8b-48f1-b005-ae52866a6e85-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.942489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.942248 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cb92f6-92cd-4265-8605-d609fd387060-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.942489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.942264 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153a9803-ee8e-41f3-92fd-6e62b33a4d87-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.942489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.942282 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fe4f43-a807-458e-a5df-7acee6e72e0a-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.943473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.943449 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf" pod "153a9803-ee8e-41f3-92fd-6e62b33a4d87" (UID: "153a9803-ee8e-41f3-92fd-6e62b33a4d87"). InnerVolumeSpecName "kube-api-access-9bbzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.943585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.943495 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6" pod "0759aa09-4f8b-48f1-b005-ae52866a6e85" (UID: "0759aa09-4f8b-48f1-b005-ae52866a6e85"). InnerVolumeSpecName "kube-api-access-psfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.943585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.943516 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4" pod "10fe4f43-a807-458e-a5df-7acee6e72e0a" (UID: "10fe4f43-a807-458e-a5df-7acee6e72e0a"). InnerVolumeSpecName "kube-api-access-xl7t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.943874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:19.943860 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf" pod "63cb92f6-92cd-4265-8605-d609fd387060" (UID: "63cb92f6-92cd-4265-8605-d609fd387060"). InnerVolumeSpecName "kube-api-access-r99hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.023590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.023559 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8"] Oct 31 00:45:20.030331 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.030307 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6a693b-29890125-b54c8"] Oct 31 00:45:20.041867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.041823 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" path="/var/lib/kubelet/pods/5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8/volumes" Oct 31 00:45:20.042382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.042360 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" path="/var/lib/kubelet/pods/b45febe8-e6e8-4230-b06c-5791e67b5ac6/volumes" Oct 31 00:45:20.043633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.043608 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r99hf\" (UniqueName: \"kubernetes.io/projected/63cb92f6-92cd-4265-8605-d609fd387060-kube-api-access-r99hf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.043733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.043639 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psfb6\" (UniqueName: \"kubernetes.io/projected/0759aa09-4f8b-48f1-b005-ae52866a6e85-kube-api-access-psfb6\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.043733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.043657 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bbzf\" (UniqueName: \"kubernetes.io/projected/153a9803-ee8e-41f3-92fd-6e62b33a4d87-kube-api-access-9bbzf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.043733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.043674 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7t4\" (UniqueName: \"kubernetes.io/projected/10fe4f43-a807-458e-a5df-7acee6e72e0a-kube-api-access-xl7t4\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.372425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.372386 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n"] Oct 31 00:45:20.376490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.376464 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890125-2j22n"] Oct 31 00:45:20.647154 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.647128 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" Oct 31 00:45:20.647654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.647125 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v" event={"ID":"63cb92f6-92cd-4265-8605-d609fd387060","Type":"ContainerDied","Data":"e44cd947eb06904986b6e6cc301c03b0aab85efafbce93ce49cd1d81815bf31a"} Oct 31 00:45:20.647654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.647344 5943 scope.go:122] "RemoveContainer" containerID="253134bbee393748d4745650580ecd803fe8c17883d0dc97884ef8a39d9d7ffc" Oct 31 00:45:20.648907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.648737 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" Oct 31 00:45:20.648907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.648746 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj" event={"ID":"153a9803-ee8e-41f3-92fd-6e62b33a4d87","Type":"ContainerDied","Data":"5526b50e87ae33b5fa07fdf0c6712dd7a7589300cff75ee94c247248f0018c4c"} Oct 31 00:45:20.654414 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.654345 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" Oct 31 00:45:20.654414 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.654365 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw" event={"ID":"0759aa09-4f8b-48f1-b005-ae52866a6e85","Type":"ContainerDied","Data":"d4eae0c71919bc8d2ba517ed5462c65b69c9a1566b4c2c12eed69f5935035dfe"} Oct 31 00:45:20.656113 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.656087 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" event={"ID":"10fe4f43-a807-458e-a5df-7acee6e72e0a","Type":"ContainerDied","Data":"c0d829b6151d08c0ee517de9dc6bec0b831b3df0048da007f105a6fd9e69255f"} Oct 31 00:45:20.656294 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.656125 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7" Oct 31 00:45:20.659261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.659242 5943 scope.go:122] "RemoveContainer" containerID="e59b06e2a951bf804c42e2d832a1ca78b9c327db935d46bce9a86163e6d47d66" Oct 31 00:45:20.668472 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.668448 5943 scope.go:122] "RemoveContainer" containerID="0e38749c1e353d0676ba725af5516a01be71581fcbc62fbdb44c8c1db70c40cc" Oct 31 00:45:20.678592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.678574 5943 scope.go:122] "RemoveContainer" containerID="96f22728aacc303a9c551cd113194114d31544547ee8c5e35388364e0223e893" Oct 31 00:45:20.855840 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.855804 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:20.972155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.972065 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj"] Oct 31 00:45:20.975911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:20.975881 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-x5ttj"] Oct 31 00:45:21.084248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:21.084216 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7"] Oct 31 00:45:21.096298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:21.096271 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-38dd3e-29890125-m5zc7"] Oct 31 00:45:21.204510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:21.204473 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v"] Oct 31 00:45:21.208192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:21.208155 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890125-dtx5v"] Oct 31 00:45:21.317228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:21.317198 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw"] Oct 31 00:45:21.320451 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:21.320426 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7ae676-29890125-6h8vw"] Oct 31 00:45:22.040657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:22.040615 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" path="/var/lib/kubelet/pods/0759aa09-4f8b-48f1-b005-ae52866a6e85/volumes" Oct 31 00:45:22.041295 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:22.041272 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" path="/var/lib/kubelet/pods/10fe4f43-a807-458e-a5df-7acee6e72e0a/volumes" Oct 31 00:45:22.041726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:22.041706 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" path="/var/lib/kubelet/pods/153a9803-ee8e-41f3-92fd-6e62b33a4d87/volumes" Oct 31 00:45:22.042471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:22.042448 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cb92f6-92cd-4265-8605-d609fd387060" path="/var/lib/kubelet/pods/63cb92f6-92cd-4265-8605-d609fd387060/volumes" Oct 31 00:45:22.042875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:22.042856 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8702a95d-da73-43c0-8243-2e0991718513" path="/var/lib/kubelet/pods/8702a95d-da73-43c0-8243-2e0991718513/volumes" Oct 31 00:45:25.327442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:25.327407 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:25.836327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:25.836280 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:26.584156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:26.584115 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:27.206833 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:27.206796 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:27.478003 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:27.477900 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:45:27.543069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:27.543029 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:27.829118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:27.829082 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:45:30.674614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:30.674568 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:31.948996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:31.948963 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:32.163035 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:32.162997 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:32.330485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:32.330446 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.217589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.217558 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf"] Oct 31 00:45:35.218093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218078 5943 cpu_manager.go:414] "containerMap: removing container" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218096 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218108 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218113 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218121 5943 cpu_manager.go:414] "containerMap: removing container" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218127 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218133 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218139 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218147 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218152 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218159 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218163 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218169 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218190 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218201 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218207 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218220 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218227 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218238 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218245 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218258 5943 cpu_manager.go:414] "containerMap: removing container" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218264 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218271 5943 cpu_manager.go:414] "containerMap: removing container" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218275 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218285 5943 cpu_manager.go:414] "containerMap: removing container" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218289 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218300 5943 cpu_manager.go:414] "containerMap: removing container" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218303 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218308 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218312 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218318 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218323 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218333 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218337 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218346 5943 cpu_manager.go:414] "containerMap: removing container" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218350 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218439 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218445 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218452 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218458 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218465 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218471 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218477 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218482 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218488 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218493 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218501 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218506 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218512 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218518 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6657ab-6e3e-4bbf-b2cf-dc322597b7e8" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218525 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8702a95d-da73-43c0-8243-2e0991718513" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218531 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:35.218521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.218536 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:35.221488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.221471 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.273758 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.273729 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:45:35.321235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.321207 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.321387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.321327 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlr44\" (UniqueName: \"kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.321454 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.321441 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.322901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.322879 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:45:35.322901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.322878 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:35.323070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.322900 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:45:35.325085 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.325068 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf"] Oct 31 00:45:35.422134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.422109 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.422296 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.422221 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.422296 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.422283 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlr44\" (UniqueName: \"kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.422904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.422831 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.424538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.424517 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.483069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.483012 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlr44\" (UniqueName: \"kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44\") pod \"maas-api-revoke-sub-2af9cdb4-wrmvf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.534361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.534323 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:35.796593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.796560 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.908995 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:35.908972 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf"] Oct 31 00:45:35.911170 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:35.911145 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63f1ca9_4fba_4544_a192_fabb3fbaedcf.slice/crio-98dbf6813b82c78059ecbce604a5c8312f71b356d844981b124afe48fa9b0900 WatchSource:0}: Error finding container 98dbf6813b82c78059ecbce604a5c8312f71b356d844981b124afe48fa9b0900: Status 404 returned error can't find the container with id 98dbf6813b82c78059ecbce604a5c8312f71b356d844981b124afe48fa9b0900 Oct 31 00:45:36.768474 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:36.768436 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" event={"ID":"e63f1ca9-4fba-4544-a192-fabb3fbaedcf","Type":"ContainerStarted","Data":"e5b126671ea85d5a3490483edb5e422f1bdcd21218c2c45e625c6b67e2b3a587"} Oct 31 00:45:36.768865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:36.768493 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" event={"ID":"e63f1ca9-4fba-4544-a192-fabb3fbaedcf","Type":"ContainerStarted","Data":"98dbf6813b82c78059ecbce604a5c8312f71b356d844981b124afe48fa9b0900"} Oct 31 00:45:36.884680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:36.884625 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" podStartSLOduration=1.8846116309999998 podStartE2EDuration="1.884611631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:36.882518697 +0000 UTC m=+2516.975808895" watchObservedRunningTime="2026-10-31 00:45:36.884611631 +0000 UTC m=+2516.977901818" Oct 31 00:45:37.345870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.345827 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.528520 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.528485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.776569 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.776531 5943 generic.go:356] "Generic (PLEG): container finished" podID="e63f1ca9-4fba-4544-a192-fabb3fbaedcf" containerID="e5b126671ea85d5a3490483edb5e422f1bdcd21218c2c45e625c6b67e2b3a587" exitCode=0 Oct 31 00:45:37.777006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.776607 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" event={"ID":"e63f1ca9-4fba-4544-a192-fabb3fbaedcf","Type":"ContainerDied","Data":"e5b126671ea85d5a3490483edb5e422f1bdcd21218c2c45e625c6b67e2b3a587"} Oct 31 00:45:37.915519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.915479 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5"] Oct 31 00:45:37.916022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916000 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:37.916127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916021 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:37.916127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916050 5943 cpu_manager.go:414] "containerMap: removing container" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:37.916127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916059 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153a9803-ee8e-41f3-92fd-6e62b33a4d87" containerName="cleanup" Oct 31 00:45:37.916127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916077 5943 cpu_manager.go:414] "containerMap: removing container" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:37.916127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916085 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:37.916333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916297 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cb92f6-92cd-4265-8605-d609fd387060" containerName="cleanup" Oct 31 00:45:37.916333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916314 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0759aa09-4f8b-48f1-b005-ae52866a6e85" containerName="cleanup" Oct 31 00:45:37.916333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916328 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fe4f43-a807-458e-a5df-7acee6e72e0a" containerName="cleanup" Oct 31 00:45:37.916418 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.916338 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45febe8-e6e8-4230-b06c-5791e67b5ac6" containerName="cleanup" Oct 31 00:45:37.919963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.919939 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:37.971976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.971945 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-31144e\"" Oct 31 00:45:37.972111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.972096 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-31144e-dockercfg-8bckm\"" Oct 31 00:45:37.978361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:37.978331 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5"] Oct 31 00:45:38.054775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.054688 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7gm\" (UniqueName: \"kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm\") pod \"payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.054958 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.054882 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.156482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.156443 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.156629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.156535 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7gm\" (UniqueName: \"kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm\") pod \"payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.157248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.157222 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.217867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.217823 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7gm\" (UniqueName: \"kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm\") pod \"payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.233663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.233640 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:38.608666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.608642 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5"] Oct 31 00:45:38.610511 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:38.610476 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766a21e4_8d3b_46f5_ad52_9514a1834483.slice/crio-d843582cf9ece80b62ef168948cdb6891523cf669f0c275b52170f9598541bbb WatchSource:0}: Error finding container d843582cf9ece80b62ef168948cdb6891523cf669f0c275b52170f9598541bbb: Status 404 returned error can't find the container with id d843582cf9ece80b62ef168948cdb6891523cf669f0c275b52170f9598541bbb Oct 31 00:45:38.784705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.784612 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" event={"ID":"766a21e4-8d3b-46f5-ad52-9514a1834483","Type":"ContainerStarted","Data":"3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1"} Oct 31 00:45:38.784705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.784668 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" event={"ID":"766a21e4-8d3b-46f5-ad52-9514a1834483","Type":"ContainerStarted","Data":"d843582cf9ece80b62ef168948cdb6891523cf669f0c275b52170f9598541bbb"} Oct 31 00:45:38.901534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.901411 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" podStartSLOduration=1.901394751 podStartE2EDuration="1.901394751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:38.899997852 +0000 UTC m=+2518.993288058" watchObservedRunningTime="2026-10-31 00:45:38.901394751 +0000 UTC m=+2518.994684936" Oct 31 00:45:38.923243 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:38.923216 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:39.068170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.068137 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca\") pod \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " Oct 31 00:45:39.068379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.068269 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44\" (UniqueName: \"kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44\") pod \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " Oct 31 00:45:39.068379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.068360 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca\") pod \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\" (UID: \"e63f1ca9-4fba-4544-a192-fabb3fbaedcf\") " Oct 31 00:45:39.068818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.068790 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca" pod "e63f1ca9-4fba-4544-a192-fabb3fbaedcf" (UID: "e63f1ca9-4fba-4544-a192-fabb3fbaedcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:39.068943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.068872 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:39.070633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.070607 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca" pod "e63f1ca9-4fba-4544-a192-fabb3fbaedcf" (UID: "e63f1ca9-4fba-4544-a192-fabb3fbaedcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:39.070728 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.070631 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44" pod "e63f1ca9-4fba-4544-a192-fabb3fbaedcf" (UID: "e63f1ca9-4fba-4544-a192-fabb3fbaedcf"). InnerVolumeSpecName "kube-api-access-nlr44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:39.169634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.169555 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:39.169634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.169585 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlr44\" (UniqueName: \"kubernetes.io/projected/e63f1ca9-4fba-4544-a192-fabb3fbaedcf-kube-api-access-nlr44\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:39.792972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.792941 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" Oct 31 00:45:39.793453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.792941 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf" event={"ID":"e63f1ca9-4fba-4544-a192-fabb3fbaedcf","Type":"ContainerDied","Data":"98dbf6813b82c78059ecbce604a5c8312f71b356d844981b124afe48fa9b0900"} Oct 31 00:45:39.793453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.793065 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98dbf6813b82c78059ecbce604a5c8312f71b356d844981b124afe48fa9b0900" Oct 31 00:45:39.793453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:39.793274 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:40.669889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:40.669838 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:40.834652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:40.834614 5943 scope.go:122] "RemoveContainer" containerID="de2c7ae8dec546085508dbd90fab8f69d7b373606d91462907e78af409093a94" Oct 31 00:45:45.452092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:45.452048 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:47.539555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:47.539519 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.382374 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:48.382338 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.815978 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:48.815898 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.422219 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:49.420406 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.695972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:49.695875 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.442688 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:50.442656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.590656 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:50.590627 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.738558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:50.738465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.803143 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:50.803117 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:45:51.786488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.786454 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4"] Oct 31 00:45:51.787062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.787031 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e63f1ca9-4fba-4544-a192-fabb3fbaedcf" containerName="revoke-keys" Oct 31 00:45:51.787062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.787053 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63f1ca9-4fba-4544-a192-fabb3fbaedcf" containerName="revoke-keys" Oct 31 00:45:51.787251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.787201 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63f1ca9-4fba-4544-a192-fabb3fbaedcf" containerName="revoke-keys" Oct 31 00:45:51.792361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.792344 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:51.844350 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.844328 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:51.844710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.844684 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:45:51.894229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.894201 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:45:51.894406 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.894291 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:45:51.896648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.896623 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4"] Oct 31 00:45:51.932549 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.932515 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvftc\" (UniqueName: \"kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:51.932749 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.932725 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:51.932793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:51.932773 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.033793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.033757 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.034016 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.033815 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.034016 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.033887 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvftc\" (UniqueName: \"kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.034539 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.034516 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.036323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.036303 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.095045 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.094976 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvftc\" (UniqueName: \"kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc\") pod \"maas-api-revoke-sub-449ae58d-cqvm4\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.105860 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.105832 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:52.489067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.489035 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4"] Oct 31 00:45:52.491730 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:45:52.491697 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021a6b2a_5157_44ba_9a15_43d47223b3ff.slice/crio-95735856cdd99277ae71deea8e5e0a598d6ae785421e101b4073f2d2d6a98308 WatchSource:0}: Error finding container 95735856cdd99277ae71deea8e5e0a598d6ae785421e101b4073f2d2d6a98308: Status 404 returned error can't find the container with id 95735856cdd99277ae71deea8e5e0a598d6ae785421e101b4073f2d2d6a98308 Oct 31 00:45:52.564357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.564327 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.878719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.878684 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" event={"ID":"021a6b2a-5157-44ba-9a15-43d47223b3ff","Type":"ContainerStarted","Data":"0867a6fb230d61134aa23669c1930b0dc4757008e5066d8c54b4932ba3212cbb"} Oct 31 00:45:52.879197 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.878742 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" event={"ID":"021a6b2a-5157-44ba-9a15-43d47223b3ff","Type":"ContainerStarted","Data":"95735856cdd99277ae71deea8e5e0a598d6ae785421e101b4073f2d2d6a98308"} Oct 31 00:45:52.995859 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:52.995810 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" podStartSLOduration=1.9957966470000001 podStartE2EDuration="1.995796647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:52.995236314 +0000 UTC m=+2533.088526522" watchObservedRunningTime="2026-10-31 00:45:52.995796647 +0000 UTC m=+2533.089086834" Oct 31 00:45:53.886225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:53.886193 5943 generic.go:356] "Generic (PLEG): container finished" podID="021a6b2a-5157-44ba-9a15-43d47223b3ff" containerID="0867a6fb230d61134aa23669c1930b0dc4757008e5066d8c54b4932ba3212cbb" exitCode=0 Oct 31 00:45:53.886642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:53.886280 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" event={"ID":"021a6b2a-5157-44ba-9a15-43d47223b3ff","Type":"ContainerDied","Data":"0867a6fb230d61134aa23669c1930b0dc4757008e5066d8c54b4932ba3212cbb"} Oct 31 00:45:53.939723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:53.939682 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.139899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:54.139813 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:55.051606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.051580 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:55.171528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.171452 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca\") pod \"021a6b2a-5157-44ba-9a15-43d47223b3ff\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " Oct 31 00:45:55.171712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.171606 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc\" (UniqueName: \"kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc\") pod \"021a6b2a-5157-44ba-9a15-43d47223b3ff\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " Oct 31 00:45:55.171712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.171663 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca\") pod \"021a6b2a-5157-44ba-9a15-43d47223b3ff\" (UID: \"021a6b2a-5157-44ba-9a15-43d47223b3ff\") " Oct 31 00:45:55.172192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.172142 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca" pod "021a6b2a-5157-44ba-9a15-43d47223b3ff" (UID: "021a6b2a-5157-44ba-9a15-43d47223b3ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:55.174338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.174315 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc" pod "021a6b2a-5157-44ba-9a15-43d47223b3ff" (UID: "021a6b2a-5157-44ba-9a15-43d47223b3ff"). InnerVolumeSpecName "kube-api-access-qvftc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:55.174502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.174482 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca" pod "021a6b2a-5157-44ba-9a15-43d47223b3ff" (UID: "021a6b2a-5157-44ba-9a15-43d47223b3ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:55.273193 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.273159 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:55.273356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.273211 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021a6b2a-5157-44ba-9a15-43d47223b3ff-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:55.273356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.273228 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvftc\" (UniqueName: \"kubernetes.io/projected/021a6b2a-5157-44ba-9a15-43d47223b3ff-kube-api-access-qvftc\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:55.707641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.707601 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:55.900379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.900349 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" Oct 31 00:45:55.900551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.900395 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4" event={"ID":"021a6b2a-5157-44ba-9a15-43d47223b3ff","Type":"ContainerDied","Data":"95735856cdd99277ae71deea8e5e0a598d6ae785421e101b4073f2d2d6a98308"} Oct 31 00:45:55.900551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:55.900470 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95735856cdd99277ae71deea8e5e0a598d6ae785421e101b4073f2d2d6a98308" Oct 31 00:45:57.477071 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:45:57.477032 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:45:57.923945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:57.923907 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.123665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.123632 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.345213 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.345164 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p"] Oct 31 00:45:58.345965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.345947 5943 cpu_manager.go:414] "containerMap: removing container" podUID="021a6b2a-5157-44ba-9a15-43d47223b3ff" containerName="revoke-keys" Oct 31 00:45:58.346022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.345968 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021a6b2a-5157-44ba-9a15-43d47223b3ff" containerName="revoke-keys" Oct 31 00:45:58.346090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.346079 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="021a6b2a-5157-44ba-9a15-43d47223b3ff" containerName="revoke-keys" Oct 31 00:45:58.350009 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.349988 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.453996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.453959 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bf38e4\"" Oct 31 00:45:58.453996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.453969 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bf38e4-dockercfg-7g6jc\"" Oct 31 00:45:58.456747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.456722 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p"] Oct 31 00:45:58.513164 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.513131 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.513598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.513311 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2w5\" (UniqueName: \"kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5\") pod \"payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.614039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.613926 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2w5\" (UniqueName: \"kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5\") pod \"payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.614039 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.614009 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.614692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.614671 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.675319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.675288 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2w5\" (UniqueName: \"kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5\") pod \"payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:58.964056 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:58.963976 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:59.342256 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:59.342231 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p"] Oct 31 00:45:59.926134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:59.926090 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" event={"ID":"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1","Type":"ContainerStarted","Data":"f85481c6db8f5a7658c542598e574d9e5eff70e4cb672a4730effddfcce71568"} Oct 31 00:45:59.926640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:59.926281 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:45:59.926640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:45:59.926316 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" event={"ID":"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1","Type":"ContainerStarted","Data":"3bc7b90ffbaafc0e1d19ee2c77426cd58d7f07873debed9ef399cb80102fee78"} Oct 31 00:46:00.043922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:00.043867 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" podStartSLOduration=2.043854797 podStartE2EDuration="2.043854797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:00.041027707 +0000 UTC m=+2540.134317914" watchObservedRunningTime="2026-10-31 00:46:00.043854797 +0000 UTC m=+2540.137145033" Oct 31 00:46:00.960562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:00.960527 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:04.739171 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:04.739132 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:46:05.767070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:05.767031 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:07.900983 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:07.900943 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs"] Oct 31 00:46:07.905124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:07.905101 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:07.920542 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:07.920515 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:07.920674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:07.920662 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:07.920736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:07.920724 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xk4\" (UniqueName: \"kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:07.957465 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:07.957429 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:46:08.006693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.006662 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:46:08.006693 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.006691 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:46:08.006911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.006662 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:08.009235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.009211 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs"] Oct 31 00:46:08.022357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.022326 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.022523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.022491 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.022612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.022596 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xk4\" (UniqueName: \"kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.023031 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.023010 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.024996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.024978 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.083645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.083610 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xk4\" (UniqueName: \"kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4\") pod \"maas-api-revoke-sub-e06782c1-rjzcs\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.218455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.218365 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:08.626907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.626879 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs"] Oct 31 00:46:08.988272 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.988233 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" event={"ID":"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d","Type":"ContainerStarted","Data":"14efe71221527ddf24dae571334376747a681249abae709b9f3283f69481fd44"} Oct 31 00:46:08.988647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:08.988284 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" event={"ID":"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d","Type":"ContainerStarted","Data":"b8e5c192bbc1f571c5a34b5e88294b3088d372efe0aa5a82a1bd19dbf60e0cdd"} Oct 31 00:46:09.112999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.112944 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" podStartSLOduration=2.112924655 podStartE2EDuration="2.112924655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:09.11142245 +0000 UTC m=+2549.204712643" watchObservedRunningTime="2026-10-31 00:46:09.112924655 +0000 UTC m=+2549.206214842" Oct 31 00:46:09.788585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.788546 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj"] Oct 31 00:46:09.792826 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.792799 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.859220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.859112 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj"] Oct 31 00:46:09.862955 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.862898 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmfh\" (UniqueName: \"kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.863132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.863078 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.863132 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.863123 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.964233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.964173 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.964416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.964242 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.964416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.964312 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmfh\" (UniqueName: \"kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.964948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.964914 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.966716 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.966690 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:09.996246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.996215 5943 generic.go:356] "Generic (PLEG): container finished" podID="f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" containerID="14efe71221527ddf24dae571334376747a681249abae709b9f3283f69481fd44" exitCode=0 Oct 31 00:46:09.996597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:09.996296 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" event={"ID":"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d","Type":"ContainerDied","Data":"14efe71221527ddf24dae571334376747a681249abae709b9f3283f69481fd44"} Oct 31 00:46:10.025010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:10.024971 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmfh\" (UniqueName: \"kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh\") pod \"maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:10.107287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:10.107206 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:10.515372 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:10.515343 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj"] Oct 31 00:46:10.516878 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:46:10.516846 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad9a9de_df68_43b9_9e9f_e6c1bff6a9be.slice/crio-d573880e79e10ae37226a7bd4723d837ee1bf258bae2908109049e3a4e0d2145 WatchSource:0}: Error finding container d573880e79e10ae37226a7bd4723d837ee1bf258bae2908109049e3a4e0d2145: Status 404 returned error can't find the container with id d573880e79e10ae37226a7bd4723d837ee1bf258bae2908109049e3a4e0d2145 Oct 31 00:46:10.854300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:10.854265 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.955455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:10.955424 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:46:11.004574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.004542 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" event={"ID":"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be","Type":"ContainerStarted","Data":"b39800d40b8229ed2c3a94275ff4cc08de71e14ba0d963ff428290851b8092a2"} Oct 31 00:46:11.004959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.004603 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" event={"ID":"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be","Type":"ContainerStarted","Data":"d573880e79e10ae37226a7bd4723d837ee1bf258bae2908109049e3a4e0d2145"} Oct 31 00:46:11.189220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.188167 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" podStartSLOduration=2.188148215 podStartE2EDuration="2.188148215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:11.186635732 +0000 UTC m=+2551.279925943" watchObservedRunningTime="2026-10-31 00:46:11.188148215 +0000 UTC m=+2551.281438403" Oct 31 00:46:11.224311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.224283 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:11.277262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.277229 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca\") pod \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " Oct 31 00:46:11.277458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.277302 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca\") pod \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " Oct 31 00:46:11.277458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.277347 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4\" (UniqueName: \"kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4\") pod \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\" (UID: \"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d\") " Oct 31 00:46:11.277680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.277652 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca" pod "f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" (UID: "f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:11.277878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.277861 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:11.279608 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.279585 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca" pod "f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" (UID: "f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:11.279702 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.279638 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4" pod "f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" (UID: "f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d"). InnerVolumeSpecName "kube-api-access-h2xk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:11.379368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.379280 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:11.379368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:11.379316 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xk4\" (UniqueName: \"kubernetes.io/projected/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d-kube-api-access-h2xk4\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:12.012357 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:12.012319 5943 generic.go:356] "Generic (PLEG): container finished" podID="aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" containerID="b39800d40b8229ed2c3a94275ff4cc08de71e14ba0d963ff428290851b8092a2" exitCode=0 Oct 31 00:46:12.012810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:12.012404 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" event={"ID":"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be","Type":"ContainerDied","Data":"b39800d40b8229ed2c3a94275ff4cc08de71e14ba0d963ff428290851b8092a2"} Oct 31 00:46:12.014134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:12.014114 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" Oct 31 00:46:12.014305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:12.014154 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs" event={"ID":"f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d","Type":"ContainerDied","Data":"b8e5c192bbc1f571c5a34b5e88294b3088d372efe0aa5a82a1bd19dbf60e0cdd"} Oct 31 00:46:12.014305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:12.014225 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8e5c192bbc1f571c5a34b5e88294b3088d372efe0aa5a82a1bd19dbf60e0cdd" Oct 31 00:46:13.196791 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.196761 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:13.301106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.301013 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca\") pod \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " Oct 31 00:46:13.301314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.301263 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca\") pod \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " Oct 31 00:46:13.301314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.301305 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh\" (UniqueName: \"kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh\") pod \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\" (UID: \"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be\") " Oct 31 00:46:13.301486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.301453 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca" pod "aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" (UID: "aad9a9de-df68-43b9-9e9f-e6c1bff6a9be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:13.301746 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.301728 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:13.303661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.303634 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh" pod "aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" (UID: "aad9a9de-df68-43b9-9e9f-e6c1bff6a9be"). InnerVolumeSpecName "kube-api-access-lfmfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:13.303762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.303665 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca" pod "aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" (UID: "aad9a9de-df68-43b9-9e9f-e6c1bff6a9be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:13.403301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.403270 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfmfh\" (UniqueName: \"kubernetes.io/projected/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-kube-api-access-lfmfh\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:13.403301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:13.403299 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:14.028440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:14.028403 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" Oct 31 00:46:14.028638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:14.028450 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj" event={"ID":"aad9a9de-df68-43b9-9e9f-e6c1bff6a9be","Type":"ContainerDied","Data":"d573880e79e10ae37226a7bd4723d837ee1bf258bae2908109049e3a4e0d2145"} Oct 31 00:46:14.028638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:14.028530 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d573880e79e10ae37226a7bd4723d837ee1bf258bae2908109049e3a4e0d2145" Oct 31 00:46:16.042503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.042461 5943 generic.go:356] "Generic (PLEG): container finished" podID="652b4620-ca12-4f36-910c-057905987c2e" containerID="aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d" exitCode=0 Oct 31 00:46:16.042967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.042525 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" event={"ID":"652b4620-ca12-4f36-910c-057905987c2e","Type":"ContainerDied","Data":"aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d"} Oct 31 00:46:16.042967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.042932 5943 scope.go:122] "RemoveContainer" containerID="aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d" Oct 31 00:46:16.044609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.044580 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:46:16.044742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.044651 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:46:16.044742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.044685 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:46:16.076101 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:16.076074 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.062048 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:17.062014 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" event={"ID":"652b4620-ca12-4f36-910c-057905987c2e","Type":"ContainerStarted","Data":"7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9"} Oct 31 00:46:18.017420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:18.017384 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:46:18.018427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:18.018395 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:46:18.018570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:18.018448 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 00:46:19.018130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:19.018089 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:46:19.018668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:19.018155 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 00:46:20.017499 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:20.017460 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:46:20.017687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:20.017517 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 00:46:21.018121 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:21.018088 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:46:21.018597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:21.018140 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 00:46:21.104260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:21.104207 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.320820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:21.320732 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:46:21.862138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:21.862106 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:22.006463 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:22.006422 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:22.017893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:22.017850 5943 patch_prober.go:29] interesting pod/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:46:22.018094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:22.017915 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 00:46:23.022565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:23.022532 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:46:23.023689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:23.023666 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:46:24.950711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.950677 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk"] Oct 31 00:46:24.951227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.951163 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" containerName="revoke-keys" Oct 31 00:46:24.951227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.951188 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" containerName="revoke-keys" Oct 31 00:46:24.951227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.951202 5943 cpu_manager.go:414] "containerMap: removing container" podUID="aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" containerName="revoke-keys" Oct 31 00:46:24.951227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.951207 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" containerName="revoke-keys" Oct 31 00:46:24.951409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.951330 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" containerName="revoke-keys" Oct 31 00:46:24.951409 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.951339 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" containerName="revoke-keys" Oct 31 00:46:24.954819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:24.954804 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.006830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.006793 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:25.007008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.006854 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:46:25.053403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.053365 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.053580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.053440 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbrs\" (UniqueName: \"kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.053631 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.053595 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.056130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.056099 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:46:25.056293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.056211 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:46:25.058711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.058688 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk"] Oct 31 00:46:25.154576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.154544 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.154756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.154649 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.154756 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.154693 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbrs\" (UniqueName: \"kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.155355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.155329 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.156993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.156969 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.216092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.216012 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbrs\" (UniqueName: \"kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs\") pod \"maas-api-revoke-sub-8f63c123-2gblk\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.267639 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.267606 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:25.645920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.645896 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk"] Oct 31 00:46:25.648221 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:46:25.648172 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c8f2be_e368_4bc7_99f7_574d0756afce.slice/crio-2a6d7811cae0aa0fcda36ddea3c9a8cd035d055680c5c49a71be18c5e44105d8 WatchSource:0}: Error finding container 2a6d7811cae0aa0fcda36ddea3c9a8cd035d055680c5c49a71be18c5e44105d8: Status 404 returned error can't find the container with id 2a6d7811cae0aa0fcda36ddea3c9a8cd035d055680c5c49a71be18c5e44105d8 Oct 31 00:46:25.849366 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:25.849335 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:26.130460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:26.130409 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" event={"ID":"62c8f2be-e368-4bc7-99f7-574d0756afce","Type":"ContainerStarted","Data":"0fe5e4d600feae3d229556e4ec8ab7b348d2cfc0501832d732ddf6d5a2001d13"} Oct 31 00:46:26.131025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:26.130506 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" event={"ID":"62c8f2be-e368-4bc7-99f7-574d0756afce","Type":"ContainerStarted","Data":"2a6d7811cae0aa0fcda36ddea3c9a8cd035d055680c5c49a71be18c5e44105d8"} Oct 31 00:46:26.247714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:26.247603 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" podStartSLOduration=2.247584942 podStartE2EDuration="2.247584942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:26.246274448 +0000 UTC m=+2566.339564653" watchObservedRunningTime="2026-10-31 00:46:26.247584942 +0000 UTC m=+2566.340875135" Oct 31 00:46:27.137989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:27.137946 5943 generic.go:356] "Generic (PLEG): container finished" podID="62c8f2be-e368-4bc7-99f7-574d0756afce" containerID="0fe5e4d600feae3d229556e4ec8ab7b348d2cfc0501832d732ddf6d5a2001d13" exitCode=0 Oct 31 00:46:27.138429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:27.137997 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" event={"ID":"62c8f2be-e368-4bc7-99f7-574d0756afce","Type":"ContainerDied","Data":"0fe5e4d600feae3d229556e4ec8ab7b348d2cfc0501832d732ddf6d5a2001d13"} Oct 31 00:46:27.476088 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:46:27.475998 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:46:28.298752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.298720 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:28.394630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.394527 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca\") pod \"62c8f2be-e368-4bc7-99f7-574d0756afce\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " Oct 31 00:46:28.394630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.394593 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs\" (UniqueName: \"kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs\") pod \"62c8f2be-e368-4bc7-99f7-574d0756afce\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " Oct 31 00:46:28.394843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.394709 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca\") pod \"62c8f2be-e368-4bc7-99f7-574d0756afce\" (UID: \"62c8f2be-e368-4bc7-99f7-574d0756afce\") " Oct 31 00:46:28.395095 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.395064 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca" pod "62c8f2be-e368-4bc7-99f7-574d0756afce" (UID: "62c8f2be-e368-4bc7-99f7-574d0756afce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:28.395298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.395272 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:28.397107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.397080 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs" pod "62c8f2be-e368-4bc7-99f7-574d0756afce" (UID: "62c8f2be-e368-4bc7-99f7-574d0756afce"). InnerVolumeSpecName "kube-api-access-kjbrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:28.397341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.397321 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca" pod "62c8f2be-e368-4bc7-99f7-574d0756afce" (UID: "62c8f2be-e368-4bc7-99f7-574d0756afce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:28.496077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.496042 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c8f2be-e368-4bc7-99f7-574d0756afce-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:28.496077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:28.496072 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbrs\" (UniqueName: \"kubernetes.io/projected/62c8f2be-e368-4bc7-99f7-574d0756afce-kube-api-access-kjbrs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:29.151583 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:29.151555 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" Oct 31 00:46:29.151734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:29.151594 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk" event={"ID":"62c8f2be-e368-4bc7-99f7-574d0756afce","Type":"ContainerDied","Data":"2a6d7811cae0aa0fcda36ddea3c9a8cd035d055680c5c49a71be18c5e44105d8"} Oct 31 00:46:29.151734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:29.151641 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6d7811cae0aa0fcda36ddea3c9a8cd035d055680c5c49a71be18c5e44105d8" Oct 31 00:46:30.482614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:30.482577 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:30.638510 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:30.638472 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:30.813737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:30.813704 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:35.491774 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.491726 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:35.759786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.759749 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk"] Oct 31 00:46:35.760460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.760445 5943 cpu_manager.go:414] "containerMap: removing container" podUID="62c8f2be-e368-4bc7-99f7-574d0756afce" containerName="revoke-keys" Oct 31 00:46:35.760514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.760463 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c8f2be-e368-4bc7-99f7-574d0756afce" containerName="revoke-keys" Oct 31 00:46:35.760623 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.760610 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c8f2be-e368-4bc7-99f7-574d0756afce" containerName="revoke-keys" Oct 31 00:46:35.763986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.763962 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.855195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.855140 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:35.855195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.855144 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:46:35.857876 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.857849 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk"] Oct 31 00:46:35.868374 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.868340 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:46:35.868374 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.868362 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:46:35.882021 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.881982 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfghs\" (UniqueName: \"kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.882334 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.882302 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.882459 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.882361 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.984375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.984331 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfghs\" (UniqueName: \"kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.984591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.984417 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.984591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.984461 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.985292 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.985256 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:35.987255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:35.987230 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:36.045360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:36.045273 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfghs\" (UniqueName: \"kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs\") pod \"maas-api-revoke-sub-df2258a4-zwfsk\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:36.082973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:36.082940 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:36.465989 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:46:36.465953 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0fd701_cede_44f9_b620_d34c8a4de30a.slice/crio-15969f23af7ca9963c743dadbf43d659bec47b48a476593d1c9072b080f6d8a6 WatchSource:0}: Error finding container 15969f23af7ca9963c743dadbf43d659bec47b48a476593d1c9072b080f6d8a6: Status 404 returned error can't find the container with id 15969f23af7ca9963c743dadbf43d659bec47b48a476593d1c9072b080f6d8a6 Oct 31 00:46:36.467337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:36.467303 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk"] Oct 31 00:46:37.203999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:37.203909 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" event={"ID":"2e0fd701-cede-44f9-b620-d34c8a4de30a","Type":"ContainerStarted","Data":"f678c0aa8a04f98aa8a489dad8a97302605997759fa7f357379779303c4c8d2b"} Oct 31 00:46:37.203999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:37.203976 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" event={"ID":"2e0fd701-cede-44f9-b620-d34c8a4de30a","Type":"ContainerStarted","Data":"15969f23af7ca9963c743dadbf43d659bec47b48a476593d1c9072b080f6d8a6"} Oct 31 00:46:37.321696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:37.321635 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" podStartSLOduration=2.321616445 podStartE2EDuration="2.321616445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:37.318738772 +0000 UTC m=+2577.412028981" watchObservedRunningTime="2026-10-31 00:46:37.321616445 +0000 UTC m=+2577.414906633" Oct 31 00:46:38.214427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:38.214392 5943 generic.go:356] "Generic (PLEG): container finished" podID="2e0fd701-cede-44f9-b620-d34c8a4de30a" containerID="f678c0aa8a04f98aa8a489dad8a97302605997759fa7f357379779303c4c8d2b" exitCode=0 Oct 31 00:46:38.214854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:38.214443 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" event={"ID":"2e0fd701-cede-44f9-b620-d34c8a4de30a","Type":"ContainerDied","Data":"f678c0aa8a04f98aa8a489dad8a97302605997759fa7f357379779303c4c8d2b"} Oct 31 00:46:39.375204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.375160 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:39.426381 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.426352 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca\") pod \"2e0fd701-cede-44f9-b620-d34c8a4de30a\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " Oct 31 00:46:39.426589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.426419 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca\") pod \"2e0fd701-cede-44f9-b620-d34c8a4de30a\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " Oct 31 00:46:39.426589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.426444 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs\" (UniqueName: \"kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs\") pod \"2e0fd701-cede-44f9-b620-d34c8a4de30a\" (UID: \"2e0fd701-cede-44f9-b620-d34c8a4de30a\") " Oct 31 00:46:39.426837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.426807 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca" pod "2e0fd701-cede-44f9-b620-d34c8a4de30a" (UID: "2e0fd701-cede-44f9-b620-d34c8a4de30a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:39.427083 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.427060 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:39.429324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.429298 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca" pod "2e0fd701-cede-44f9-b620-d34c8a4de30a" (UID: "2e0fd701-cede-44f9-b620-d34c8a4de30a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:39.429444 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.429333 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs" pod "2e0fd701-cede-44f9-b620-d34c8a4de30a" (UID: "2e0fd701-cede-44f9-b620-d34c8a4de30a"). InnerVolumeSpecName "kube-api-access-hfghs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:39.527899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.527870 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0fd701-cede-44f9-b620-d34c8a4de30a-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:39.527899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:39.527897 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfghs\" (UniqueName: \"kubernetes.io/projected/2e0fd701-cede-44f9-b620-d34c8a4de30a-kube-api-access-hfghs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:40.228719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:40.228679 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" event={"ID":"2e0fd701-cede-44f9-b620-d34c8a4de30a","Type":"ContainerDied","Data":"15969f23af7ca9963c743dadbf43d659bec47b48a476593d1c9072b080f6d8a6"} Oct 31 00:46:40.228719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:40.228706 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk" Oct 31 00:46:40.228930 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:40.228734 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15969f23af7ca9963c743dadbf43d659bec47b48a476593d1c9072b080f6d8a6" Oct 31 00:46:40.347874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:40.347780 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:40.889578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:40.889545 5943 scope.go:122] "RemoveContainer" containerID="fd3657c40f73a16d5d0b9b82dbb0591b5e46901b37edb24e2d4bf1fe2cc8aea5" Oct 31 00:46:45.098059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:45.098020 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:46.162382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:46.162352 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.862932 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:49.862875 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:51.883997 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.883962 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86"] Oct 31 00:46:51.884498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.884484 5943 cpu_manager.go:414] "containerMap: removing container" podUID="2e0fd701-cede-44f9-b620-d34c8a4de30a" containerName="revoke-keys" Oct 31 00:46:51.884543 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.884498 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0fd701-cede-44f9-b620-d34c8a4de30a" containerName="revoke-keys" Oct 31 00:46:51.884615 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.884605 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0fd701-cede-44f9-b620-d34c8a4de30a" containerName="revoke-keys" Oct 31 00:46:51.889337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.889317 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:51.943820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.943764 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:46:51.967060 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.967021 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:51.967269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.967096 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:46:51.967269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.967105 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:46:51.977008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.976966 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:51.977172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.977133 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:51.977310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.977296 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82v8\" (UniqueName: \"kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:51.995579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:51.995552 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86"] Oct 31 00:46:52.078896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.078858 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82v8\" (UniqueName: \"kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.079092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.078913 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.079092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.078972 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.079636 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.079611 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.081491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.081471 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.140210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.140121 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82v8\" (UniqueName: \"kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8\") pod \"maas-api-revoke-sub-fe2c2a56-djx86\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.203142 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.203114 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:52.579762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.579732 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86"] Oct 31 00:46:52.581480 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:46:52.581455 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28ac4da_7e56_4c4c_b59a_7f2d7fce1786.slice/crio-08e051cb30ac4c7d6e92dfa98eff3e9957278fde4d04cb77e06ed05a9fe53dc5 WatchSource:0}: Error finding container 08e051cb30ac4c7d6e92dfa98eff3e9957278fde4d04cb77e06ed05a9fe53dc5: Status 404 returned error can't find the container with id 08e051cb30ac4c7d6e92dfa98eff3e9957278fde4d04cb77e06ed05a9fe53dc5 Oct 31 00:46:52.905374 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:52.905341 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:53.311613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:53.311574 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" event={"ID":"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786","Type":"ContainerStarted","Data":"cccf8c611398ba2f264a52db4af1a121f2be457ae58142770170d41b0cd6a458"} Oct 31 00:46:53.311779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:53.311632 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" event={"ID":"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786","Type":"ContainerStarted","Data":"08e051cb30ac4c7d6e92dfa98eff3e9957278fde4d04cb77e06ed05a9fe53dc5"} Oct 31 00:46:53.426935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:53.426885 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" podStartSLOduration=2.42687184 podStartE2EDuration="2.42687184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:53.426812546 +0000 UTC m=+2593.520102770" watchObservedRunningTime="2026-10-31 00:46:53.42687184 +0000 UTC m=+2593.520162027" Oct 31 00:46:54.320790 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:54.320754 5943 generic.go:356] "Generic (PLEG): container finished" podID="d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" containerID="cccf8c611398ba2f264a52db4af1a121f2be457ae58142770170d41b0cd6a458" exitCode=0 Oct 31 00:46:54.321205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:54.320841 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" event={"ID":"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786","Type":"ContainerDied","Data":"cccf8c611398ba2f264a52db4af1a121f2be457ae58142770170d41b0cd6a458"} Oct 31 00:46:54.668610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:54.668491 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.476798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.476771 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:55.623920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.623831 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8\" (UniqueName: \"kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8\") pod \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " Oct 31 00:46:55.623920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.623887 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca\") pod \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " Oct 31 00:46:55.624143 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.624045 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca\") pod \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\" (UID: \"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786\") " Oct 31 00:46:55.624321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.624289 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca" pod "d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" (UID: "d28ac4da-7e56-4c4c-b59a-7f2d7fce1786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:55.624683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.624658 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:55.626229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.626203 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca" pod "d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" (UID: "d28ac4da-7e56-4c4c-b59a-7f2d7fce1786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:55.626229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.626218 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8" pod "d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" (UID: "d28ac4da-7e56-4c4c-b59a-7f2d7fce1786"). InnerVolumeSpecName "kube-api-access-g82v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:55.725482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.725447 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:55.725482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:55.725476 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82v8\" (UniqueName: \"kubernetes.io/projected/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786-kube-api-access-g82v8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:56.336283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:56.336208 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" Oct 31 00:46:56.336429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:56.336207 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86" event={"ID":"d28ac4da-7e56-4c4c-b59a-7f2d7fce1786","Type":"ContainerDied","Data":"08e051cb30ac4c7d6e92dfa98eff3e9957278fde4d04cb77e06ed05a9fe53dc5"} Oct 31 00:46:56.336429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:56.336326 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08e051cb30ac4c7d6e92dfa98eff3e9957278fde4d04cb77e06ed05a9fe53dc5" Oct 31 00:46:57.476035 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:46:57.475995 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:46:59.295003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:59.294969 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:59.489936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:59.489897 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:59.637640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:46:59.637547 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:00.798468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:00.798436 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:47:01.224078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:01.223985 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.650397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.650369 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg"] Oct 31 00:47:03.650884 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.650871 5943 cpu_manager.go:414] "containerMap: removing container" podUID="d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" containerName="revoke-keys" Oct 31 00:47:03.650933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.650885 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" containerName="revoke-keys" Oct 31 00:47:03.650981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.650972 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" containerName="revoke-keys" Oct 31 00:47:03.654503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.654487 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.706672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.706646 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:03.721291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.721262 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnv7\" (UniqueName: \"kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.721446 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.721318 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.721515 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.721489 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.756214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.756161 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:03.756401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.756381 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:03.756458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.756422 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:47:03.758971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.758949 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg"] Oct 31 00:47:03.822327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.822286 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnv7\" (UniqueName: \"kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.822526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.822352 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.822526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.822441 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.823010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.822983 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.825165 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.825141 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.883605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.883565 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnv7\" (UniqueName: \"kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7\") pod \"maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:03.967912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:03.967829 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:04.346361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:04.346318 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg"] Oct 31 00:47:04.347247 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:04.347212 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafc1e3e_b673_4dc6_85d5_f98a3e22cfab.slice/crio-259bcc6ecd62c8ba54a7059a806fef1d73be3275106c221f33ba2fab09547c3c WatchSource:0}: Error finding container 259bcc6ecd62c8ba54a7059a806fef1d73be3275106c221f33ba2fab09547c3c: Status 404 returned error can't find the container with id 259bcc6ecd62c8ba54a7059a806fef1d73be3275106c221f33ba2fab09547c3c Oct 31 00:47:04.400037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:04.400007 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" event={"ID":"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab","Type":"ContainerStarted","Data":"259bcc6ecd62c8ba54a7059a806fef1d73be3275106c221f33ba2fab09547c3c"} Oct 31 00:47:04.656160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:04.656124 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:05.408044 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:05.408000 5943 generic.go:356] "Generic (PLEG): container finished" podID="cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" containerID="7def9eb15eb00457bb2f4e96d9c43170d29d67dadd61f4f948c0252a16568a3a" exitCode=0 Oct 31 00:47:05.408264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:05.408097 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" event={"ID":"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab","Type":"ContainerDied","Data":"7def9eb15eb00457bb2f4e96d9c43170d29d67dadd61f4f948c0252a16568a3a"} Oct 31 00:47:06.565488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.565462 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:06.655443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.655406 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca\") pod \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " Oct 31 00:47:06.655617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.655515 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7\" (UniqueName: \"kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7\") pod \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " Oct 31 00:47:06.655617 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.655580 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca\") pod \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\" (UID: \"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab\") " Oct 31 00:47:06.655783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.655760 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca" pod "cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" (UID: "cafc1e3e-b673-4dc6-85d5-f98a3e22cfab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:06.655907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.655894 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.657804 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.657786 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca" pod "cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" (UID: "cafc1e3e-b673-4dc6-85d5-f98a3e22cfab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:06.657996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.657968 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7" pod "cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" (UID: "cafc1e3e-b673-4dc6-85d5-f98a3e22cfab"). InnerVolumeSpecName "kube-api-access-gtnv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:06.757019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.756984 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnv7\" (UniqueName: \"kubernetes.io/projected/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-kube-api-access-gtnv7\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.757019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:06.757014 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:07.422562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.422529 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" Oct 31 00:47:07.422562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.422546 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg" event={"ID":"cafc1e3e-b673-4dc6-85d5-f98a3e22cfab","Type":"ContainerDied","Data":"259bcc6ecd62c8ba54a7059a806fef1d73be3275106c221f33ba2fab09547c3c"} Oct 31 00:47:07.422773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.422589 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259bcc6ecd62c8ba54a7059a806fef1d73be3275106c221f33ba2fab09547c3c" Oct 31 00:47:07.804673 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.804639 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj"] Oct 31 00:47:07.805289 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.805271 5943 cpu_manager.go:414] "containerMap: removing container" podUID="cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" containerName="revoke-keys" Oct 31 00:47:07.805371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.805290 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" containerName="revoke-keys" Oct 31 00:47:07.805418 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.805397 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" containerName="revoke-keys" Oct 31 00:47:07.808880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.808861 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.864750 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.864681 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:07.870468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.870436 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.870685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.870664 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.870741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.870709 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqh8\" (UniqueName: \"kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.913502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.913464 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:07.913502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.913495 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:07.913709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.913521 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:47:07.916425 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.916402 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj"] Oct 31 00:47:07.972100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.972070 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.972280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.972133 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqh8\" (UniqueName: \"kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.972280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.972234 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.972805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.972777 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:07.974724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:07.974703 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:08.033359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:08.033334 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqh8\" (UniqueName: \"kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8\") pod \"maas-api-revoke-sub-515a7592-m6fqj\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:08.124774 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:08.124687 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:08.500258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:08.500232 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj"] Oct 31 00:47:08.503679 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:08.503646 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c3eb58_c529_4782_934b_47ef69770a57.slice/crio-8b6120facc4566f95b81212da95a4d793761a604bb55e8e49557c2365b462e1e WatchSource:0}: Error finding container 8b6120facc4566f95b81212da95a4d793761a604bb55e8e49557c2365b462e1e: Status 404 returned error can't find the container with id 8b6120facc4566f95b81212da95a4d793761a604bb55e8e49557c2365b462e1e Oct 31 00:47:09.439270 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:09.439226 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" event={"ID":"b8c3eb58-c529-4782-934b-47ef69770a57","Type":"ContainerStarted","Data":"bf1fae566aab469c35a300f146a9d53284d586a4ec4eea7c611e5d3dc617a75c"} Oct 31 00:47:09.439629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:09.439330 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" event={"ID":"b8c3eb58-c529-4782-934b-47ef69770a57","Type":"ContainerStarted","Data":"8b6120facc4566f95b81212da95a4d793761a604bb55e8e49557c2365b462e1e"} Oct 31 00:47:09.780033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:09.779999 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.447741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.447706 5943 generic.go:356] "Generic (PLEG): container finished" podID="b8c3eb58-c529-4782-934b-47ef69770a57" containerID="bf1fae566aab469c35a300f146a9d53284d586a4ec4eea7c611e5d3dc617a75c" exitCode=0 Oct 31 00:47:10.448169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.447792 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" event={"ID":"b8c3eb58-c529-4782-934b-47ef69770a57","Type":"ContainerDied","Data":"bf1fae566aab469c35a300f146a9d53284d586a4ec4eea7c611e5d3dc617a75c"} Oct 31 00:47:10.596133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.596111 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:10.707221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.707120 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8\" (UniqueName: \"kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8\") pod \"b8c3eb58-c529-4782-934b-47ef69770a57\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " Oct 31 00:47:10.707221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.707163 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca\") pod \"b8c3eb58-c529-4782-934b-47ef69770a57\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " Oct 31 00:47:10.707470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.707233 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca\") pod \"b8c3eb58-c529-4782-934b-47ef69770a57\" (UID: \"b8c3eb58-c529-4782-934b-47ef69770a57\") " Oct 31 00:47:10.707576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.707552 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca" pod "b8c3eb58-c529-4782-934b-47ef69770a57" (UID: "b8c3eb58-c529-4782-934b-47ef69770a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.707740 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.707725 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.709467 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.709443 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8" pod "b8c3eb58-c529-4782-934b-47ef69770a57" (UID: "b8c3eb58-c529-4782-934b-47ef69770a57"). InnerVolumeSpecName "kube-api-access-smqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.709540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.709493 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca" pod "b8c3eb58-c529-4782-934b-47ef69770a57" (UID: "b8c3eb58-c529-4782-934b-47ef69770a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.808796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.808766 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smqh8\" (UniqueName: \"kubernetes.io/projected/b8c3eb58-c529-4782-934b-47ef69770a57-kube-api-access-smqh8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.808796 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:10.808792 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3eb58-c529-4782-934b-47ef69770a57-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:11.455894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:11.455856 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" Oct 31 00:47:11.456324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:11.455896 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj" event={"ID":"b8c3eb58-c529-4782-934b-47ef69770a57","Type":"ContainerDied","Data":"8b6120facc4566f95b81212da95a4d793761a604bb55e8e49557c2365b462e1e"} Oct 31 00:47:11.456324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:11.455949 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b6120facc4566f95b81212da95a4d793761a604bb55e8e49557c2365b462e1e" Oct 31 00:47:14.441938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.441899 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p"] Oct 31 00:47:14.442566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.442549 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b8c3eb58-c529-4782-934b-47ef69770a57" containerName="revoke-keys" Oct 31 00:47:14.442635 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.442568 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c3eb58-c529-4782-934b-47ef69770a57" containerName="revoke-keys" Oct 31 00:47:14.442705 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.442677 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c3eb58-c529-4782-934b-47ef69770a57" containerName="revoke-keys" Oct 31 00:47:14.446209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.446172 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.501883 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.501851 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:14.502421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.502389 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:14.502421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.502390 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:14.502659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.502429 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:47:14.506209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.506143 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p"] Oct 31 00:47:14.553699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.553657 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.553699 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.553700 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.553915 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.553897 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqpp\" (UniqueName: \"kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.654767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.654726 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqpp\" (UniqueName: \"kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.654940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.654813 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.654940 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.654834 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.655456 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.655439 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.657441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.657419 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.715900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.715835 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqpp\" (UniqueName: \"kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp\") pod \"maas-api-revoke-sub-aa3f3f39-88l9p\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:14.761571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:14.761540 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:15.137941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.137917 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p"] Oct 31 00:47:15.139520 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:15.139492 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d9e02f_53a2_4725_a451_a457a0984683.slice/crio-319ca964c7437bd7932d5b732c836f786ca2701386df80af8ba3c8f80f4869df WatchSource:0}: Error finding container 319ca964c7437bd7932d5b732c836f786ca2701386df80af8ba3c8f80f4869df: Status 404 returned error can't find the container with id 319ca964c7437bd7932d5b732c836f786ca2701386df80af8ba3c8f80f4869df Oct 31 00:47:15.161090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.161061 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:15.484426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.484383 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" event={"ID":"76d9e02f-53a2-4725-a451-a457a0984683","Type":"ContainerStarted","Data":"bd66e620123c94033a5ce53c1982d8ab4e3739c003958c2e1fef91779d797953"} Oct 31 00:47:15.484946 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.484455 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" event={"ID":"76d9e02f-53a2-4725-a451-a457a0984683","Type":"ContainerStarted","Data":"319ca964c7437bd7932d5b732c836f786ca2701386df80af8ba3c8f80f4869df"} Oct 31 00:47:15.602126 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.602063 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" podStartSLOduration=1.6020485089999998 podStartE2EDuration="1.602048509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:15.599555905 +0000 UTC m=+2615.692846115" watchObservedRunningTime="2026-10-31 00:47:15.602048509 +0000 UTC m=+2615.695338692" Oct 31 00:47:15.609768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.609737 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:15.759839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:15.759799 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:16.492579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:16.492497 5943 generic.go:356] "Generic (PLEG): container finished" podID="76d9e02f-53a2-4725-a451-a457a0984683" containerID="bd66e620123c94033a5ce53c1982d8ab4e3739c003958c2e1fef91779d797953" exitCode=0 Oct 31 00:47:16.492945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:16.492582 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" event={"ID":"76d9e02f-53a2-4725-a451-a457a0984683","Type":"ContainerDied","Data":"bd66e620123c94033a5ce53c1982d8ab4e3739c003958c2e1fef91779d797953"} Oct 31 00:47:17.647471 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.647447 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:17.791921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.791894 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca\") pod \"76d9e02f-53a2-4725-a451-a457a0984683\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " Oct 31 00:47:17.792098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.791946 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp\" (UniqueName: \"kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp\") pod \"76d9e02f-53a2-4725-a451-a457a0984683\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " Oct 31 00:47:17.792098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.791978 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca\") pod \"76d9e02f-53a2-4725-a451-a457a0984683\" (UID: \"76d9e02f-53a2-4725-a451-a457a0984683\") " Oct 31 00:47:17.792407 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.792374 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca" pod "76d9e02f-53a2-4725-a451-a457a0984683" (UID: "76d9e02f-53a2-4725-a451-a457a0984683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:17.792567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.792548 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.794313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.794289 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp" pod "76d9e02f-53a2-4725-a451-a457a0984683" (UID: "76d9e02f-53a2-4725-a451-a457a0984683"). InnerVolumeSpecName "kube-api-access-pnqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.794692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.794672 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca" pod "76d9e02f-53a2-4725-a451-a457a0984683" (UID: "76d9e02f-53a2-4725-a451-a457a0984683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:17.894095 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.894060 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqpp\" (UniqueName: \"kubernetes.io/projected/76d9e02f-53a2-4725-a451-a457a0984683-kube-api-access-pnqpp\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.894095 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:17.894094 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d9e02f-53a2-4725-a451-a457a0984683-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:18.506874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:18.506845 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" Oct 31 00:47:18.506874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:18.506854 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p" event={"ID":"76d9e02f-53a2-4725-a451-a457a0984683","Type":"ContainerDied","Data":"319ca964c7437bd7932d5b732c836f786ca2701386df80af8ba3c8f80f4869df"} Oct 31 00:47:18.507118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:18.506900 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="319ca964c7437bd7932d5b732c836f786ca2701386df80af8ba3c8f80f4869df" Oct 31 00:47:20.038703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:20.038669 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:20.559868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:20.559831 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:20.890331 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:20.890224 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:24.847111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:24.847069 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:25.768265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.768231 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg"] Oct 31 00:47:25.768802 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.768783 5943 cpu_manager.go:414] "containerMap: removing container" podUID="76d9e02f-53a2-4725-a451-a457a0984683" containerName="revoke-keys" Oct 31 00:47:25.768903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.768805 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d9e02f-53a2-4725-a451-a457a0984683" containerName="revoke-keys" Oct 31 00:47:25.769030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.769015 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d9e02f-53a2-4725-a451-a457a0984683" containerName="revoke-keys" Oct 31 00:47:25.772864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.772842 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788269 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788438 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788326 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788474 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788589 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5j4\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788634 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788657 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788695 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788675 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788734 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.788861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.788795 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.844921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.844885 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg"] Oct 31 00:47:25.889646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.889607 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.889714 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5j4\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.889806 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.889884 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.889915 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.889947 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890026 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890144 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890206 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890591 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890618 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890656 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.890944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.890918 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.891387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.891351 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.893470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.893449 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.893602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.893489 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.951373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.951337 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:25.951527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:25.951384 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5j4\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:26.087349 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:26.087261 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:26.526817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:26.526792 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg"] Oct 31 00:47:26.528516 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:26.528486 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7ee1fe_c3bb_49a3_b1be_d01b07dcee4d.slice/crio-d6c5258f3dc6350b1acf7a414a2892668c44910db6a0e2d5805d7d650a1a1c90 WatchSource:0}: Error finding container d6c5258f3dc6350b1acf7a414a2892668c44910db6a0e2d5805d7d650a1a1c90: Status 404 returned error can't find the container with id d6c5258f3dc6350b1acf7a414a2892668c44910db6a0e2d5805d7d650a1a1c90 Oct 31 00:47:26.530952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:26.530919 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:47:26.531041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:26.530985 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:47:26.531041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:26.531014 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:47:26.561766 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:26.561735 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" event={"ID":"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d","Type":"ContainerStarted","Data":"d6c5258f3dc6350b1acf7a414a2892668c44910db6a0e2d5805d7d650a1a1c90"} Oct 31 00:47:27.482999 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:47:27.482962 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:47:27.570878 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:27.570841 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" event={"ID":"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d","Type":"ContainerStarted","Data":"d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf"} Oct 31 00:47:27.802333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:27.802272 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podStartSLOduration=2.8022583660000002 podStartE2EDuration="2.802258366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:27.752836447 +0000 UTC m=+2627.846126635" watchObservedRunningTime="2026-10-31 00:47:27.802258366 +0000 UTC m=+2627.895548552" Oct 31 00:47:28.088172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:28.088085 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:28.089904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:28.089871 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:47:28.090026 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:28.089940 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:47:29.087863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.087831 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:47:29.088397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.087892 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:47:29.687214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.687167 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.769743 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.769709 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc"] Oct 31 00:47:29.773991 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.773969 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.833775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.833628 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:29.833775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.833747 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:29.834138 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.834118 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:29.859630 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.859586 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkd9r\" (UniqueName: \"kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.859831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.859700 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.859831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.859747 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.883075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.883047 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:47:29.885863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.885840 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc"] Oct 31 00:47:29.961020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.960935 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.961020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.960979 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.961228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.961135 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkd9r\" (UniqueName: \"kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.961734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.961713 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:29.963506 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:29.963489 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:30.022597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:30.022563 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkd9r\" (UniqueName: \"kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r\") pod \"maas-api-revoke-sub-6fca6de8-5m9mc\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:30.088022 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:30.087984 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:47:30.088468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:30.088041 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:47:30.088514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:30.088496 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:30.487509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:30.487483 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc"] Oct 31 00:47:30.489908 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:30.489884 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce8b42f_fccc_456a_bdd7_b582e081ddd5.slice/crio-3bd23e95535c83f75479ac9c850154076c183733b2d159ce11cf9b1f7a49cea1 WatchSource:0}: Error finding container 3bd23e95535c83f75479ac9c850154076c183733b2d159ce11cf9b1f7a49cea1: Status 404 returned error can't find the container with id 3bd23e95535c83f75479ac9c850154076c183733b2d159ce11cf9b1f7a49cea1 Oct 31 00:47:30.591271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:30.591230 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" event={"ID":"fce8b42f-fccc-456a-bdd7-b582e081ddd5","Type":"ContainerStarted","Data":"3bd23e95535c83f75479ac9c850154076c183733b2d159ce11cf9b1f7a49cea1"} Oct 31 00:47:31.087670 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:31.087635 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:47:31.087871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:31.087684 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:47:31.599237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:31.599159 5943 generic.go:356] "Generic (PLEG): container finished" podID="fce8b42f-fccc-456a-bdd7-b582e081ddd5" containerID="efa13f51fa60252135fba1edb7cedc2dafa7a2ec4d62be94f50ad0b88ecd8d08" exitCode=0 Oct 31 00:47:31.599237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:31.599223 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" event={"ID":"fce8b42f-fccc-456a-bdd7-b582e081ddd5","Type":"ContainerDied","Data":"efa13f51fa60252135fba1edb7cedc2dafa7a2ec4d62be94f50ad0b88ecd8d08"} Oct 31 00:47:32.087692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.087657 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:47:32.087869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.087731 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:47:32.747942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.747912 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:32.795619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.795574 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r\" (UniqueName: \"kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r\") pod \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " Oct 31 00:47:32.795811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.795643 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca\") pod \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " Oct 31 00:47:32.795811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.795760 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca\") pod \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\" (UID: \"fce8b42f-fccc-456a-bdd7-b582e081ddd5\") " Oct 31 00:47:32.796114 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.796084 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca" pod "fce8b42f-fccc-456a-bdd7-b582e081ddd5" (UID: "fce8b42f-fccc-456a-bdd7-b582e081ddd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:32.796327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.796279 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:32.797903 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.797875 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca" pod "fce8b42f-fccc-456a-bdd7-b582e081ddd5" (UID: "fce8b42f-fccc-456a-bdd7-b582e081ddd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:32.798002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.797923 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r" pod "fce8b42f-fccc-456a-bdd7-b582e081ddd5" (UID: "fce8b42f-fccc-456a-bdd7-b582e081ddd5"). InnerVolumeSpecName "kube-api-access-zkd9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:32.897337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.897250 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkd9r\" (UniqueName: \"kubernetes.io/projected/fce8b42f-fccc-456a-bdd7-b582e081ddd5-kube-api-access-zkd9r\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:32.897337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:32.897283 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce8b42f-fccc-456a-bdd7-b582e081ddd5-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:33.088464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:33.088417 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:47:33.088638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:33.088488 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:47:33.616553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:33.616518 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" event={"ID":"fce8b42f-fccc-456a-bdd7-b582e081ddd5","Type":"ContainerDied","Data":"3bd23e95535c83f75479ac9c850154076c183733b2d159ce11cf9b1f7a49cea1"} Oct 31 00:47:33.616762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:33.616575 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bd23e95535c83f75479ac9c850154076c183733b2d159ce11cf9b1f7a49cea1" Oct 31 00:47:33.616762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:33.616639 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc" Oct 31 00:47:34.098722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:34.098688 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:34.099229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:34.098990 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:34.099754 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:34.099736 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 00:47:34.551288 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:34.551246 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.336105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:39.336071 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.496706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:39.496674 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:47:40.917002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:40.916917 5943 scope.go:122] "RemoveContainer" containerID="6c4f6b85b8494065c5fa736e4177ba43a117402c1c8fee3300e66edae88ff154" Oct 31 00:47:40.934701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:40.934675 5943 scope.go:122] "RemoveContainer" containerID="c827c27b1a72f393613f2305b663cf5ea88121d7cf3322dab03c58e0304113e4" Oct 31 00:47:43.936251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:43.936210 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.143874 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:45.143838 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.631990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:45.631944 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.883013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:45.882923 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:46.093873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:46.093827 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:46.327256 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:46.327223 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg"] Oct 31 00:47:46.376577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:46.376544 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38dd3e-4vtpg"] Oct 31 00:47:46.824394 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:46.824358 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:48.042422 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:48.042390 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafc1e3e-b673-4dc6-85d5-f98a3e22cfab" path="/var/lib/kubelet/pods/cafc1e3e-b673-4dc6-85d5-f98a3e22cfab/volumes" Oct 31 00:47:50.419806 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.419772 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp"] Oct 31 00:47:50.420301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.420286 5943 cpu_manager.go:414] "containerMap: removing container" podUID="fce8b42f-fccc-456a-bdd7-b582e081ddd5" containerName="revoke-keys" Oct 31 00:47:50.420358 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.420302 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce8b42f-fccc-456a-bdd7-b582e081ddd5" containerName="revoke-keys" Oct 31 00:47:50.420433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.420424 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce8b42f-fccc-456a-bdd7-b582e081ddd5" containerName="revoke-keys" Oct 31 00:47:50.424994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.424978 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.478934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.478879 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:50.479111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.479059 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:50.528574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.528546 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:50.528761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.528742 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:47:50.531312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.531291 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp"] Oct 31 00:47:50.608109 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.608080 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.608293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.608131 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.608293 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.608265 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcxv\" (UniqueName: \"kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.709798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.709716 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcxv\" (UniqueName: \"kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.709973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.709953 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.710027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.710013 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.710643 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.710612 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.712523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.712494 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:50.770933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:50.770901 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcxv\" (UniqueName: \"kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:51.006344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.006308 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:51.037964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.037939 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:51.427873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.427846 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp"] Oct 31 00:47:51.429913 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:51.429879 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4a543f_12fb_4f60_a4fd_e1372c615e36.slice/crio-886332005e5e6b8886b71d90036cbf5433d7280746b15c7c4ebe7223fc7387f3 WatchSource:0}: Error finding container 886332005e5e6b8886b71d90036cbf5433d7280746b15c7c4ebe7223fc7387f3: Status 404 returned error can't find the container with id 886332005e5e6b8886b71d90036cbf5433d7280746b15c7c4ebe7223fc7387f3 Oct 31 00:47:51.765885 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.765853 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" event={"ID":"4a4a543f-12fb-4f60-a4fd-e1372c615e36","Type":"ContainerStarted","Data":"f9e1c93bf10bd429e45f6784d967453fe6750e17b403fd6a6aa28183b28dc8b2"} Oct 31 00:47:51.766019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.765919 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" event={"ID":"4a4a543f-12fb-4f60-a4fd-e1372c615e36","Type":"ContainerStarted","Data":"886332005e5e6b8886b71d90036cbf5433d7280746b15c7c4ebe7223fc7387f3"} Oct 31 00:47:51.768404 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.768384 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv"] Oct 31 00:47:51.772436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.772422 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:51.831985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.831955 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv"] Oct 31 00:47:51.884291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.884241 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" podStartSLOduration=1.884225029 podStartE2EDuration="1.884225029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:51.882805371 +0000 UTC m=+2651.976095575" watchObservedRunningTime="2026-10-31 00:47:51.884225029 +0000 UTC m=+2651.977515216" Oct 31 00:47:51.929457 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.929428 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:51.929609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.929467 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:51.929609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:51.929560 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkj7x\" (UniqueName: \"kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.030882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.030834 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.031078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.030931 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkj7x\" (UniqueName: \"kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.031135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.031107 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.031503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.031481 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.033681 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.033658 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.091640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.091602 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkj7x\" (UniqueName: \"kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x\") pod \"maas-api-revoke-sub-d59b5af2-qhknv\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.386591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.386561 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:52.767905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.767871 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv"] Oct 31 00:47:52.769735 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:52.769698 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9646978_d03a_4013_ab69_9fbb4a5aeda7.slice/crio-d9915a11b751c89ff67b45683fc2ab968e5a2960b9caf311d69234c92e3e5174 WatchSource:0}: Error finding container d9915a11b751c89ff67b45683fc2ab968e5a2960b9caf311d69234c92e3e5174: Status 404 returned error can't find the container with id d9915a11b751c89ff67b45683fc2ab968e5a2960b9caf311d69234c92e3e5174 Oct 31 00:47:52.782281 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.782248 5943 generic.go:356] "Generic (PLEG): container finished" podID="4a4a543f-12fb-4f60-a4fd-e1372c615e36" containerID="f9e1c93bf10bd429e45f6784d967453fe6750e17b403fd6a6aa28183b28dc8b2" exitCode=0 Oct 31 00:47:52.782396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:52.782333 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" event={"ID":"4a4a543f-12fb-4f60-a4fd-e1372c615e36","Type":"ContainerDied","Data":"f9e1c93bf10bd429e45f6784d967453fe6750e17b403fd6a6aa28183b28dc8b2"} Oct 31 00:47:53.045546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.045511 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:53.790558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.790526 5943 generic.go:356] "Generic (PLEG): container finished" podID="e9646978-d03a-4013-ab69-9fbb4a5aeda7" containerID="282fc84a2cfdec25219218f5cfa3cc30d2738286961917e3dbb319491b486115" exitCode=0 Oct 31 00:47:53.791062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.790675 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" event={"ID":"e9646978-d03a-4013-ab69-9fbb4a5aeda7","Type":"ContainerDied","Data":"282fc84a2cfdec25219218f5cfa3cc30d2738286961917e3dbb319491b486115"} Oct 31 00:47:53.791062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.790748 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" event={"ID":"e9646978-d03a-4013-ab69-9fbb4a5aeda7","Type":"ContainerStarted","Data":"d9915a11b751c89ff67b45683fc2ab968e5a2960b9caf311d69234c92e3e5174"} Oct 31 00:47:53.945742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.945714 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:53.954206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.954157 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca\") pod \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " Oct 31 00:47:53.954312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.954296 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv\" (UniqueName: \"kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv\") pod \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " Oct 31 00:47:53.954384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.954366 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca\") pod \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\" (UID: \"4a4a543f-12fb-4f60-a4fd-e1372c615e36\") " Oct 31 00:47:53.954592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.954565 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca" pod "4a4a543f-12fb-4f60-a4fd-e1372c615e36" (UID: "4a4a543f-12fb-4f60-a4fd-e1372c615e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:53.954956 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.954895 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:53.956571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.956547 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca" pod "4a4a543f-12fb-4f60-a4fd-e1372c615e36" (UID: "4a4a543f-12fb-4f60-a4fd-e1372c615e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:53.956571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:53.956549 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv" pod "4a4a543f-12fb-4f60-a4fd-e1372c615e36" (UID: "4a4a543f-12fb-4f60-a4fd-e1372c615e36"). InnerVolumeSpecName "kube-api-access-lxcxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:54.056724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.056687 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxcxv\" (UniqueName: \"kubernetes.io/projected/4a4a543f-12fb-4f60-a4fd-e1372c615e36-kube-api-access-lxcxv\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:54.056724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.056723 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4a543f-12fb-4f60-a4fd-e1372c615e36-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:54.270473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.270442 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh"] Oct 31 00:47:54.270994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.270981 5943 cpu_manager.go:414] "containerMap: removing container" podUID="4a4a543f-12fb-4f60-a4fd-e1372c615e36" containerName="revoke-keys" Oct 31 00:47:54.271041 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.270997 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4a543f-12fb-4f60-a4fd-e1372c615e36" containerName="revoke-keys" Oct 31 00:47:54.271083 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.271075 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4a543f-12fb-4f60-a4fd-e1372c615e36" containerName="revoke-keys" Oct 31 00:47:54.274525 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.274511 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.332801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.332774 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh"] Oct 31 00:47:54.360571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.360539 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.360751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.360719 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.360751 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.360749 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4sh\" (UniqueName: \"kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.361102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.361080 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:54.461709 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.461680 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.461917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.461782 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.461917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.461802 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4sh\" (UniqueName: \"kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.462497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.462475 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.464602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.464577 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.522757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.522691 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4sh\" (UniqueName: \"kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh\") pod \"maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.588269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.588241 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:54.798591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.798507 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" Oct 31 00:47:54.798591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.798539 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp" event={"ID":"4a4a543f-12fb-4f60-a4fd-e1372c615e36","Type":"ContainerDied","Data":"886332005e5e6b8886b71d90036cbf5433d7280746b15c7c4ebe7223fc7387f3"} Oct 31 00:47:54.799025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.798607 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886332005e5e6b8886b71d90036cbf5433d7280746b15c7c4ebe7223fc7387f3" Oct 31 00:47:54.951105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.951081 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:54.965752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.965726 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh"] Oct 31 00:47:54.967608 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.967576 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca\") pod \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " Oct 31 00:47:54.967738 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:47:54.967583 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee7ab44_286a_4cb6_bf03_7a2ff05eb2e3.slice/crio-277e6da5640a10d546f379b57de32ada60491d4ce037d112c1eed44ddd62ecfa WatchSource:0}: Error finding container 277e6da5640a10d546f379b57de32ada60491d4ce037d112c1eed44ddd62ecfa: Status 404 returned error can't find the container with id 277e6da5640a10d546f379b57de32ada60491d4ce037d112c1eed44ddd62ecfa Oct 31 00:47:54.967738 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.967686 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x\" (UniqueName: \"kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x\") pod \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " Oct 31 00:47:54.967922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.967904 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca\") pod \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\" (UID: \"e9646978-d03a-4013-ab69-9fbb4a5aeda7\") " Oct 31 00:47:54.968336 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.968311 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca" pod "e9646978-d03a-4013-ab69-9fbb4a5aeda7" (UID: "e9646978-d03a-4013-ab69-9fbb4a5aeda7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:54.969961 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.969940 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x" pod "e9646978-d03a-4013-ab69-9fbb4a5aeda7" (UID: "e9646978-d03a-4013-ab69-9fbb4a5aeda7"). InnerVolumeSpecName "kube-api-access-vkj7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:54.970240 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:54.970221 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca" pod "e9646978-d03a-4013-ab69-9fbb4a5aeda7" (UID: "e9646978-d03a-4013-ab69-9fbb4a5aeda7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:55.069841 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.069740 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:55.069841 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.069768 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9646978-d03a-4013-ab69-9fbb4a5aeda7-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:55.069841 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.069777 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkj7x\" (UniqueName: \"kubernetes.io/projected/e9646978-d03a-4013-ab69-9fbb4a5aeda7-kube-api-access-vkj7x\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:55.806663 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.806624 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" event={"ID":"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3","Type":"ContainerStarted","Data":"b416ca7d0eaf51cb361ceb0c7e252ff5cd54ccadf895ae435dabf0e5ff432bb9"} Oct 31 00:47:55.807230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.806682 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" event={"ID":"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3","Type":"ContainerStarted","Data":"277e6da5640a10d546f379b57de32ada60491d4ce037d112c1eed44ddd62ecfa"} Oct 31 00:47:55.808115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.808096 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" Oct 31 00:47:55.808203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.808140 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv" event={"ID":"e9646978-d03a-4013-ab69-9fbb4a5aeda7","Type":"ContainerDied","Data":"d9915a11b751c89ff67b45683fc2ab968e5a2960b9caf311d69234c92e3e5174"} Oct 31 00:47:55.808245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.808200 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9915a11b751c89ff67b45683fc2ab968e5a2960b9caf311d69234c92e3e5174" Oct 31 00:47:55.928703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:55.928604 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" podStartSLOduration=1.928587848 podStartE2EDuration="1.928587848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:55.927442228 +0000 UTC m=+2656.020732435" watchObservedRunningTime="2026-10-31 00:47:55.928587848 +0000 UTC m=+2656.021878035" Oct 31 00:47:56.136934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:56.136844 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:56.816207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:56.816164 5943 generic.go:356] "Generic (PLEG): container finished" podID="fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" containerID="b416ca7d0eaf51cb361ceb0c7e252ff5cd54ccadf895ae435dabf0e5ff432bb9" exitCode=0 Oct 31 00:47:56.816613 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:56.816253 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" event={"ID":"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3","Type":"ContainerDied","Data":"b416ca7d0eaf51cb361ceb0c7e252ff5cd54ccadf895ae435dabf0e5ff432bb9"} Oct 31 00:47:57.476762 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:47:57.476723 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:47:57.986240 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:57.986207 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:58.110664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.110581 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca\") pod \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " Oct 31 00:47:58.110842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.110779 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca\") pod \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " Oct 31 00:47:58.110924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.110876 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh\" (UniqueName: \"kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh\") pod \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\" (UID: \"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3\") " Oct 31 00:47:58.111285 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.111252 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca" pod "fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" (UID: "fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:58.111448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.111419 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.113491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.113463 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh" pod "fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" (UID: "fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3"). InnerVolumeSpecName "kube-api-access-7v4sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:58.113984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.113961 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca" pod "fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" (UID: "fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:58.212263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.212228 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4sh\" (UniqueName: \"kubernetes.io/projected/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-kube-api-access-7v4sh\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.212263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.212258 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.837755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.837724 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" Oct 31 00:47:58.837966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.837761 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh" event={"ID":"fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3","Type":"ContainerDied","Data":"277e6da5640a10d546f379b57de32ada60491d4ce037d112c1eed44ddd62ecfa"} Oct 31 00:47:58.837966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:47:58.837811 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277e6da5640a10d546f379b57de32ada60491d4ce037d112c1eed44ddd62ecfa" Oct 31 00:48:01.302652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:01.302612 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:02.370534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:02.370499 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:02.570869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:02.570829 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:06.374335 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:06.374300 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:07.113811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:07.113775 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:08.099108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:08.099072 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm"] Oct 31 00:48:08.469282 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:08.469196 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:08.655875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:08.655836 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:09.023507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.023468 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld"] Oct 31 00:48:09.024043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.024026 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e9646978-d03a-4013-ab69-9fbb4a5aeda7" containerName="revoke-keys" Oct 31 00:48:09.024122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.024045 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9646978-d03a-4013-ab69-9fbb4a5aeda7" containerName="revoke-keys" Oct 31 00:48:09.024122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.024067 5943 cpu_manager.go:414] "containerMap: removing container" podUID="fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" containerName="revoke-keys" Oct 31 00:48:09.024122 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.024073 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" containerName="revoke-keys" Oct 31 00:48:09.024249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.024156 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" containerName="revoke-keys" Oct 31 00:48:09.024249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.024171 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9646978-d03a-4013-ab69-9fbb4a5aeda7" containerName="revoke-keys" Oct 31 00:48:09.027509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.027494 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.081963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.081935 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:09.082140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.082017 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:09.088236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.088214 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:09.088397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.088256 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:48:09.092114 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.092072 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld"] Oct 31 00:48:09.160643 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.160604 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vsm\" (UniqueName: \"kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.161150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.160710 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.161150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.160930 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.262593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.262547 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vsm\" (UniqueName: \"kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.262593 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.262596 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.262824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.262701 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.263594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.263559 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.265219 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.265164 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.324129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.324045 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vsm\" (UniqueName: \"kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.341424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.341396 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:09.725914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.725884 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld"] Oct 31 00:48:09.727749 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:48:09.727722 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2431e4fb_c3f9_471a_8f8e_f8a8fc810508.slice/crio-abe4b7e70ed780f0f804d85631ef1544cc162e2bd54c3e67a7183ffdd5d64ad8 WatchSource:0}: Error finding container abe4b7e70ed780f0f804d85631ef1544cc162e2bd54c3e67a7183ffdd5d64ad8: Status 404 returned error can't find the container with id abe4b7e70ed780f0f804d85631ef1544cc162e2bd54c3e67a7183ffdd5d64ad8 Oct 31 00:48:09.921600 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:09.921565 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" event={"ID":"2431e4fb-c3f9-471a-8f8e-f8a8fc810508","Type":"ContainerStarted","Data":"abe4b7e70ed780f0f804d85631ef1544cc162e2bd54c3e67a7183ffdd5d64ad8"} Oct 31 00:48:10.929256 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:10.929220 5943 generic.go:356] "Generic (PLEG): container finished" podID="2431e4fb-c3f9-471a-8f8e-f8a8fc810508" containerID="84a27dc3844283ad8ea104ffba936bf3a3cd206c442fd8f9cfb5e87e159281c5" exitCode=0 Oct 31 00:48:10.929682 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:10.929305 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" event={"ID":"2431e4fb-c3f9-471a-8f8e-f8a8fc810508","Type":"ContainerDied","Data":"84a27dc3844283ad8ea104ffba936bf3a3cd206c442fd8f9cfb5e87e159281c5"} Oct 31 00:48:11.462827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.462792 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t"] Oct 31 00:48:11.466785 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.466763 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.525580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.525549 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t"] Oct 31 00:48:11.555875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.555841 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.593620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.593587 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.593859 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.593829 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlk2\" (UniqueName: \"kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.593937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.593872 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.694775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.694745 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlk2\" (UniqueName: \"kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.694971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.694784 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.694971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.694899 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.695714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.695671 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.697680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.697645 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.756211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.756150 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlk2\" (UniqueName: \"kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2\") pod \"maas-api-revoke-sub-ccb6f9a5-pqv8t\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:11.780324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:11.780298 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:12.072504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.072475 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:12.159211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.159167 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t"] Oct 31 00:48:12.160581 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:48:12.160553 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b97fa9_f534_4947_896a_147fb4f91e4f.slice/crio-98d06460799fa49a4f5441a14e4d342542c33a17e98b45bb402a3e1ed31faa00 WatchSource:0}: Error finding container 98d06460799fa49a4f5441a14e4d342542c33a17e98b45bb402a3e1ed31faa00: Status 404 returned error can't find the container with id 98d06460799fa49a4f5441a14e4d342542c33a17e98b45bb402a3e1ed31faa00 Oct 31 00:48:12.200091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.200057 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm\" (UniqueName: \"kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm\") pod \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " Oct 31 00:48:12.200206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.200155 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca\") pod \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " Oct 31 00:48:12.200280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.200261 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca\") pod \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\" (UID: \"2431e4fb-c3f9-471a-8f8e-f8a8fc810508\") " Oct 31 00:48:12.200724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.200660 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca" pod "2431e4fb-c3f9-471a-8f8e-f8a8fc810508" (UID: "2431e4fb-c3f9-471a-8f8e-f8a8fc810508"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:12.202291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.202264 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca" pod "2431e4fb-c3f9-471a-8f8e-f8a8fc810508" (UID: "2431e4fb-c3f9-471a-8f8e-f8a8fc810508"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:12.202479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.202454 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm" pod "2431e4fb-c3f9-471a-8f8e-f8a8fc810508" (UID: "2431e4fb-c3f9-471a-8f8e-f8a8fc810508"). InnerVolumeSpecName "kube-api-access-j6vsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:12.302458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.302418 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6vsm\" (UniqueName: \"kubernetes.io/projected/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-kube-api-access-j6vsm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.302458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.302451 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.302458 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.302462 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2431e4fb-c3f9-471a-8f8e-f8a8fc810508-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.945653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.945625 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" Oct 31 00:48:12.945821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.945661 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld" event={"ID":"2431e4fb-c3f9-471a-8f8e-f8a8fc810508","Type":"ContainerDied","Data":"abe4b7e70ed780f0f804d85631ef1544cc162e2bd54c3e67a7183ffdd5d64ad8"} Oct 31 00:48:12.945821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.945755 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abe4b7e70ed780f0f804d85631ef1544cc162e2bd54c3e67a7183ffdd5d64ad8" Oct 31 00:48:12.947868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.947839 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" event={"ID":"36b97fa9-f534-4947-896a-147fb4f91e4f","Type":"ContainerStarted","Data":"b38185a1e5145cfbfe37eee60fd27b04ba3a7e64cf93765e68e1e6aa8dfe67b1"} Oct 31 00:48:12.948033 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:12.947889 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" event={"ID":"36b97fa9-f534-4947-896a-147fb4f91e4f","Type":"ContainerStarted","Data":"98d06460799fa49a4f5441a14e4d342542c33a17e98b45bb402a3e1ed31faa00"} Oct 31 00:48:13.065820 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.065761 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" podStartSLOduration=2.065745587 podStartE2EDuration="2.065745587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:13.063639788 +0000 UTC m=+2673.156930008" watchObservedRunningTime="2026-10-31 00:48:13.065745587 +0000 UTC m=+2673.159035774" Oct 31 00:48:13.099447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.099382 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" podUID="84e9b1db-5573-41cd-a69f-035dd1854a73" containerName="payload-processing" containerID="cri-o://bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1" gracePeriod=25 Oct 31 00:48:13.366920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.366890 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:48:13.417027 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.416993 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv\" (UniqueName: \"kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv\") pod \"84e9b1db-5573-41cd-a69f-035dd1854a73\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " Oct 31 00:48:13.417246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.417118 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume\") pod \"84e9b1db-5573-41cd-a69f-035dd1854a73\" (UID: \"84e9b1db-5573-41cd-a69f-035dd1854a73\") " Oct 31 00:48:13.417559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.417535 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume" pod "84e9b1db-5573-41cd-a69f-035dd1854a73" (UID: "84e9b1db-5573-41cd-a69f-035dd1854a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:13.417696 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.417683 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9b1db-5573-41cd-a69f-035dd1854a73-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:13.419335 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.419313 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv" pod "84e9b1db-5573-41cd-a69f-035dd1854a73" (UID: "84e9b1db-5573-41cd-a69f-035dd1854a73"). InnerVolumeSpecName "kube-api-access-92rwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:13.518938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.518906 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92rwv\" (UniqueName: \"kubernetes.io/projected/84e9b1db-5573-41cd-a69f-035dd1854a73-kube-api-access-92rwv\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:13.956405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.956301 5943 generic.go:356] "Generic (PLEG): container finished" podID="84e9b1db-5573-41cd-a69f-035dd1854a73" containerID="bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1" exitCode=0 Oct 31 00:48:13.956405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.956345 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" Oct 31 00:48:13.956405 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.956394 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" event={"ID":"84e9b1db-5573-41cd-a69f-035dd1854a73","Type":"ContainerDied","Data":"bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1"} Oct 31 00:48:13.956690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.956450 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm" event={"ID":"84e9b1db-5573-41cd-a69f-035dd1854a73","Type":"ContainerDied","Data":"b2e7c0fdc2b4dde94d3efc0bf5f40c3e3211150756832ab0962fd5132f4c304d"} Oct 31 00:48:13.956690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.956475 5943 scope.go:122] "RemoveContainer" containerID="bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1" Oct 31 00:48:13.958506 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.958482 5943 generic.go:356] "Generic (PLEG): container finished" podID="36b97fa9-f534-4947-896a-147fb4f91e4f" containerID="b38185a1e5145cfbfe37eee60fd27b04ba3a7e64cf93765e68e1e6aa8dfe67b1" exitCode=0 Oct 31 00:48:13.958616 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.958534 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" event={"ID":"36b97fa9-f534-4947-896a-147fb4f91e4f","Type":"ContainerDied","Data":"b38185a1e5145cfbfe37eee60fd27b04ba3a7e64cf93765e68e1e6aa8dfe67b1"} Oct 31 00:48:13.967440 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.967422 5943 scope.go:122] "RemoveContainer" containerID="bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1" Oct 31 00:48:13.967725 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:48:13.967707 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1\": container with ID starting with bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1 not found: ID does not exist" containerID="bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1" Oct 31 00:48:13.967775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:13.967734 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1"} err="failed to get container status \"bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1\": rpc error: code = NotFound desc = could not find container \"bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1\": container with ID starting with bdb3d1db630e62178e8796aea24f370201abd1c2ac6b170dd48b80d3caec73c1 not found: ID does not exist" Oct 31 00:48:14.235283 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:14.235173 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm"] Oct 31 00:48:14.239470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:14.239427 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ef8827-5686ffd4f7-tf9tm"] Oct 31 00:48:15.114752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.114727 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:15.137975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.137946 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2\" (UniqueName: \"kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2\") pod \"36b97fa9-f534-4947-896a-147fb4f91e4f\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " Oct 31 00:48:15.137975 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.137978 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca\") pod \"36b97fa9-f534-4947-896a-147fb4f91e4f\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " Oct 31 00:48:15.138247 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.138039 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca\") pod \"36b97fa9-f534-4947-896a-147fb4f91e4f\" (UID: \"36b97fa9-f534-4947-896a-147fb4f91e4f\") " Oct 31 00:48:15.138403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.138379 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca" pod "36b97fa9-f534-4947-896a-147fb4f91e4f" (UID: "36b97fa9-f534-4947-896a-147fb4f91e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:15.138609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.138590 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:15.140316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.140292 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca" pod "36b97fa9-f534-4947-896a-147fb4f91e4f" (UID: "36b97fa9-f534-4947-896a-147fb4f91e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:15.140403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.140342 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2" pod "36b97fa9-f534-4947-896a-147fb4f91e4f" (UID: "36b97fa9-f534-4947-896a-147fb4f91e4f"). InnerVolumeSpecName "kube-api-access-stlk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:15.239747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.239660 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlk2\" (UniqueName: \"kubernetes.io/projected/36b97fa9-f534-4947-896a-147fb4f91e4f-kube-api-access-stlk2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:15.239747 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.239686 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b97fa9-f534-4947-896a-147fb4f91e4f-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:15.974278 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.974248 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" Oct 31 00:48:15.974278 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.974250 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t" event={"ID":"36b97fa9-f534-4947-896a-147fb4f91e4f","Type":"ContainerDied","Data":"98d06460799fa49a4f5441a14e4d342542c33a17e98b45bb402a3e1ed31faa00"} Oct 31 00:48:15.974508 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:15.974347 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d06460799fa49a4f5441a14e4d342542c33a17e98b45bb402a3e1ed31faa00" Oct 31 00:48:16.041786 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:16.041744 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e9b1db-5573-41cd-a69f-035dd1854a73" path="/var/lib/kubelet/pods/84e9b1db-5573-41cd-a69f-035dd1854a73/volumes" Oct 31 00:48:16.362541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:16.362503 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.645332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:16.645243 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.801690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:16.801654 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:20.133038 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:20.132999 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:20.280058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:20.280025 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:20.564420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:20.564382 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.202760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:21.202709 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.347827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:21.347790 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.697700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:21.697663 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:22.057312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:22.057258 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:22.337806 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:22.337715 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:22.589065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:22.588983 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj"] Oct 31 00:48:22.637103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:22.637060 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-87b0f8-5wqwj"] Oct 31 00:48:23.067967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:23.067933 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.328872 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:23.328769 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.607883 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:23.607805 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.754330 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:23.754289 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.925441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:23.925356 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp"] Oct 31 00:48:23.974763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:23.974735 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ae676-r8cqp"] Oct 31 00:48:24.041886 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:24.041856 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4a543f-12fb-4f60-a4fd-e1372c615e36" path="/var/lib/kubelet/pods/4a4a543f-12fb-4f60-a4fd-e1372c615e36/volumes" Oct 31 00:48:24.042323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:24.042307 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad9a9de-df68-43b9-9e9f-e6c1bff6a9be" path="/var/lib/kubelet/pods/aad9a9de-df68-43b9-9e9f-e6c1bff6a9be/volumes" Oct 31 00:48:24.312910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:24.312861 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.139094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139059 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj"] Oct 31 00:48:27.139647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139630 5943 cpu_manager.go:414] "containerMap: removing container" podUID="2431e4fb-c3f9-471a-8f8e-f8a8fc810508" containerName="revoke-keys" Oct 31 00:48:27.139694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139651 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2431e4fb-c3f9-471a-8f8e-f8a8fc810508" containerName="revoke-keys" Oct 31 00:48:27.139694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139667 5943 cpu_manager.go:414] "containerMap: removing container" podUID="84e9b1db-5573-41cd-a69f-035dd1854a73" containerName="payload-processing" Oct 31 00:48:27.139694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139673 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e9b1db-5573-41cd-a69f-035dd1854a73" containerName="payload-processing" Oct 31 00:48:27.139694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139682 5943 cpu_manager.go:414] "containerMap: removing container" podUID="36b97fa9-f534-4947-896a-147fb4f91e4f" containerName="revoke-keys" Oct 31 00:48:27.139694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139687 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b97fa9-f534-4947-896a-147fb4f91e4f" containerName="revoke-keys" Oct 31 00:48:27.139851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139778 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b97fa9-f534-4947-896a-147fb4f91e4f" containerName="revoke-keys" Oct 31 00:48:27.139851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139796 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="2431e4fb-c3f9-471a-8f8e-f8a8fc810508" containerName="revoke-keys" Oct 31 00:48:27.139851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.139804 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e9b1db-5573-41cd-a69f-035dd1854a73" containerName="payload-processing" Oct 31 00:48:27.144558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.144538 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.201227 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.201196 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj"] Oct 31 00:48:27.247504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.247472 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:27.247504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.247485 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:27.247504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.247509 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:27.247967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.247551 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:48:27.285430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.285402 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.285564 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.285457 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.285605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.285588 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fl7\" (UniqueName: \"kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.387080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.387046 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fl7\" (UniqueName: \"kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.387304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.387252 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.387356 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.387309 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.388004 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.387971 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.389826 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.389771 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.448070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.448040 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fl7\" (UniqueName: \"kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7\") pod \"maas-api-revoke-sub-03cc1d26-wnnnj\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.458987 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.458950 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:27.480278 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:48:27.480242 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:48:27.835823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:27.835788 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj"] Oct 31 00:48:27.836937 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:48:27.836893 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1bf2f6_ba69_4f1f_8e8a_2ecf350b28b9.slice/crio-2f5edc408065e6c6915584e8c75d3a6876e0cbb97bbf18822be50e243a9d4693 WatchSource:0}: Error finding container 2f5edc408065e6c6915584e8c75d3a6876e0cbb97bbf18822be50e243a9d4693: Status 404 returned error can't find the container with id 2f5edc408065e6c6915584e8c75d3a6876e0cbb97bbf18822be50e243a9d4693 Oct 31 00:48:28.058898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:28.058866 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" event={"ID":"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9","Type":"ContainerStarted","Data":"2f5edc408065e6c6915584e8c75d3a6876e0cbb97bbf18822be50e243a9d4693"} Oct 31 00:48:28.220834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:28.220802 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.866168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:28.866133 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:29.070248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:29.068343 5943 generic.go:356] "Generic (PLEG): container finished" podID="dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" containerID="37a090b1a8c1d9410d357d8e251b7968670730948d46935d6f5787a0295a2c1e" exitCode=0 Oct 31 00:48:29.070248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:29.068427 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" event={"ID":"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9","Type":"ContainerDied","Data":"37a090b1a8c1d9410d357d8e251b7968670730948d46935d6f5787a0295a2c1e"} Oct 31 00:48:30.223913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.223885 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:30.323346 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.323310 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca\") pod \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " Oct 31 00:48:30.323563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.323367 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca\") pod \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " Oct 31 00:48:30.323563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.323424 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7\" (UniqueName: \"kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7\") pod \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\" (UID: \"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9\") " Oct 31 00:48:30.323793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.323760 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca" pod "dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" (UID: "dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:30.324042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.324014 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:30.325993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.325924 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7" pod "dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" (UID: "dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9"). InnerVolumeSpecName "kube-api-access-57fl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:30.326103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.326074 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca" pod "dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" (UID: "dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:30.425686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.425645 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:30.425686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.425677 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57fl7\" (UniqueName: \"kubernetes.io/projected/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9-kube-api-access-57fl7\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:30.636654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.636560 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.831236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.831203 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.994723 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:30.994644 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:31.091478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.089715 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" event={"ID":"dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9","Type":"ContainerDied","Data":"2f5edc408065e6c6915584e8c75d3a6876e0cbb97bbf18822be50e243a9d4693"} Oct 31 00:48:31.091478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.089781 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5edc408065e6c6915584e8c75d3a6876e0cbb97bbf18822be50e243a9d4693" Oct 31 00:48:31.091478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.089865 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj" Oct 31 00:48:31.855865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.855830 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq"] Oct 31 00:48:31.856427 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.856411 5943 cpu_manager.go:414] "containerMap: removing container" podUID="dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" containerName="revoke-keys" Oct 31 00:48:31.856475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.856428 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" containerName="revoke-keys" Oct 31 00:48:31.856579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.856569 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" containerName="revoke-keys" Oct 31 00:48:31.860135 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.860121 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:31.916362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.916318 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:48:31.916558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.916377 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:31.916558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.916425 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:31.942264 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.942226 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:31.942434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.942364 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:31.942555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.942537 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqxw\" (UniqueName: \"kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:31.965561 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.965528 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:31.968512 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:31.968491 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq"] Oct 31 00:48:32.043249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.043222 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqxw\" (UniqueName: \"kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.043449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.043279 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.043449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.043315 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.043909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.043891 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.045715 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.045697 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.118230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.118121 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqxw\" (UniqueName: \"kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw\") pod \"maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.174447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.174417 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:32.552394 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.552370 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq"] Oct 31 00:48:32.554416 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:48:32.554387 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eea6a98_acca_49cd_9d34_06490c9c1670.slice/crio-8c81e877c93a09127fb77b5a9e0de517532e45f506abba4bbc239e756c6dcdd5 WatchSource:0}: Error finding container 8c81e877c93a09127fb77b5a9e0de517532e45f506abba4bbc239e756c6dcdd5: Status 404 returned error can't find the container with id 8c81e877c93a09127fb77b5a9e0de517532e45f506abba4bbc239e756c6dcdd5 Oct 31 00:48:32.973459 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:32.973432 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:48:33.104775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:33.104673 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" event={"ID":"6eea6a98-acca-49cd-9d34-06490c9c1670","Type":"ContainerStarted","Data":"06a51186e09c3ecd2246c6d8df2299d787e828971d952890d1ed74010d9bdfd6"} Oct 31 00:48:33.104775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:33.104745 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" event={"ID":"6eea6a98-acca-49cd-9d34-06490c9c1670","Type":"ContainerStarted","Data":"8c81e877c93a09127fb77b5a9e0de517532e45f506abba4bbc239e756c6dcdd5"} Oct 31 00:48:33.221648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:33.221591 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" podStartSLOduration=2.221576889 podStartE2EDuration="2.221576889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:33.218813407 +0000 UTC m=+2693.312103588" watchObservedRunningTime="2026-10-31 00:48:33.221576889 +0000 UTC m=+2693.314867076" Oct 31 00:48:33.994498 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:33.994464 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:34.123761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:34.123718 5943 generic.go:356] "Generic (PLEG): container finished" podID="6eea6a98-acca-49cd-9d34-06490c9c1670" containerID="06a51186e09c3ecd2246c6d8df2299d787e828971d952890d1ed74010d9bdfd6" exitCode=0 Oct 31 00:48:34.123939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:34.123793 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" event={"ID":"6eea6a98-acca-49cd-9d34-06490c9c1670","Type":"ContainerDied","Data":"06a51186e09c3ecd2246c6d8df2299d787e828971d952890d1ed74010d9bdfd6"} Oct 31 00:48:35.381280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.381239 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:35.408385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.408302 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw\" (UniqueName: \"kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw\") pod \"6eea6a98-acca-49cd-9d34-06490c9c1670\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " Oct 31 00:48:35.408541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.408397 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca\") pod \"6eea6a98-acca-49cd-9d34-06490c9c1670\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " Oct 31 00:48:35.408541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.408501 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca\") pod \"6eea6a98-acca-49cd-9d34-06490c9c1670\" (UID: \"6eea6a98-acca-49cd-9d34-06490c9c1670\") " Oct 31 00:48:35.408807 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.408779 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca" pod "6eea6a98-acca-49cd-9d34-06490c9c1670" (UID: "6eea6a98-acca-49cd-9d34-06490c9c1670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:35.409029 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.409012 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:35.411154 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.411127 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca" pod "6eea6a98-acca-49cd-9d34-06490c9c1670" (UID: "6eea6a98-acca-49cd-9d34-06490c9c1670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:35.411274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.411244 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw" pod "6eea6a98-acca-49cd-9d34-06490c9c1670" (UID: "6eea6a98-acca-49cd-9d34-06490c9c1670"). InnerVolumeSpecName "kube-api-access-wgqxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:35.509971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.509942 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqxw\" (UniqueName: \"kubernetes.io/projected/6eea6a98-acca-49cd-9d34-06490c9c1670-kube-api-access-wgqxw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:35.509971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:35.509969 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eea6a98-acca-49cd-9d34-06490c9c1670-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.139782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:36.139751 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" Oct 31 00:48:36.139956 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:36.139794 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq" event={"ID":"6eea6a98-acca-49cd-9d34-06490c9c1670","Type":"ContainerDied","Data":"8c81e877c93a09127fb77b5a9e0de517532e45f506abba4bbc239e756c6dcdd5"} Oct 31 00:48:36.139956 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:36.139842 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c81e877c93a09127fb77b5a9e0de517532e45f506abba4bbc239e756c6dcdd5" Oct 31 00:48:38.982502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:38.982468 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:40.996255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:40.996221 5943 scope.go:122] "RemoveContainer" containerID="10691af3a1c1eb1aade603dedc413b1d336f79025fa1876b2435b724b4cf7211" Oct 31 00:48:41.007337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:41.007318 5943 scope.go:122] "RemoveContainer" containerID="a01290c79d3f1141cc4999ccba30f4b37102131c668050ac9ddbde628c310372" Oct 31 00:48:42.513540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.513510 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf"] Oct 31 00:48:42.514030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.514016 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6eea6a98-acca-49cd-9d34-06490c9c1670" containerName="revoke-keys" Oct 31 00:48:42.514107 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.514032 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eea6a98-acca-49cd-9d34-06490c9c1670" containerName="revoke-keys" Oct 31 00:48:42.514159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.514150 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eea6a98-acca-49cd-9d34-06490c9c1670" containerName="revoke-keys" Oct 31 00:48:42.518755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.518735 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.571889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.571806 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:42.602066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.602023 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.602275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.602095 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wk7\" (UniqueName: \"kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.602275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.602261 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.621314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.621289 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:42.621314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.621300 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:42.621497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.621461 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:48:42.624503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.624482 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf"] Oct 31 00:48:42.703981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.703944 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.704162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.703993 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wk7\" (UniqueName: \"kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.704162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.704042 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.704656 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.704623 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.706584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.706561 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.764884 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.764810 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wk7\" (UniqueName: \"kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7\") pod \"maas-api-revoke-sub-aa3f536d-crlwf\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:42.837625 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:42.837593 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:43.213828 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:43.213798 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf"] Oct 31 00:48:43.214855 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:48:43.214830 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f37fe2f_7e84_4cf4_9878_51da1dfc8973.slice/crio-090e076d36b43c962a5d44c130ac1fc36e37d08a813a2ef7a222dee7b35bab4f WatchSource:0}: Error finding container 090e076d36b43c962a5d44c130ac1fc36e37d08a813a2ef7a222dee7b35bab4f: Status 404 returned error can't find the container with id 090e076d36b43c962a5d44c130ac1fc36e37d08a813a2ef7a222dee7b35bab4f Oct 31 00:48:43.432827 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:48:43.432787 5943 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f37fe2f_7e84_4cf4_9878_51da1dfc8973.slice/crio-d1614d36de8f4a956bc57145c4bbf8b7ecda64aabc561b4820b1ad24d98137b3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:48:43.432827 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:48:43.432807 5943 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f37fe2f_7e84_4cf4_9878_51da1dfc8973.slice/crio-d1614d36de8f4a956bc57145c4bbf8b7ecda64aabc561b4820b1ad24d98137b3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:48:43.593248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:43.593155 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.205203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:44.205136 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" event={"ID":"9f37fe2f-7e84-4cf4-9878-51da1dfc8973","Type":"ContainerStarted","Data":"d1614d36de8f4a956bc57145c4bbf8b7ecda64aabc561b4820b1ad24d98137b3"} Oct 31 00:48:44.205411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:44.205390 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" event={"ID":"9f37fe2f-7e84-4cf4-9878-51da1dfc8973","Type":"ContainerStarted","Data":"090e076d36b43c962a5d44c130ac1fc36e37d08a813a2ef7a222dee7b35bab4f"} Oct 31 00:48:44.220861 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:44.220835 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5"] Oct 31 00:48:44.331606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:44.331550 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" podStartSLOduration=2.331535635 podStartE2EDuration="2.331535635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:44.330125502 +0000 UTC m=+2704.423415690" watchObservedRunningTime="2026-10-31 00:48:44.331535635 +0000 UTC m=+2704.424825822" Oct 31 00:48:44.588047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:44.588010 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.732599 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:44.732567 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.180914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.180883 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.213900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.213867 5943 generic.go:356] "Generic (PLEG): container finished" podID="9f37fe2f-7e84-4cf4-9878-51da1dfc8973" containerID="d1614d36de8f4a956bc57145c4bbf8b7ecda64aabc561b4820b1ad24d98137b3" exitCode=0 Oct 31 00:48:45.214096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.213950 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" event={"ID":"9f37fe2f-7e84-4cf4-9878-51da1dfc8973","Type":"ContainerDied","Data":"d1614d36de8f4a956bc57145c4bbf8b7ecda64aabc561b4820b1ad24d98137b3"} Oct 31 00:48:45.439914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.439821 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.584839 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.584799 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.767408 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.767363 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld"] Oct 31 00:48:45.770046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:45.770026 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a693b-7stld"] Oct 31 00:48:46.041265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.041159 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2431e4fb-c3f9-471a-8f8e-f8a8fc810508" path="/var/lib/kubelet/pods/2431e4fb-c3f9-471a-8f8e-f8a8fc810508/volumes" Oct 31 00:48:46.236436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.236400 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:46.377591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.377567 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:46.449714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.449683 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca\") pod \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " Oct 31 00:48:46.449882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.449809 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7\" (UniqueName: \"kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7\") pod \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " Oct 31 00:48:46.449882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.449867 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca\") pod \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\" (UID: \"9f37fe2f-7e84-4cf4-9878-51da1dfc8973\") " Oct 31 00:48:46.450069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.450050 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca" pod "9f37fe2f-7e84-4cf4-9878-51da1dfc8973" (UID: "9f37fe2f-7e84-4cf4-9878-51da1dfc8973"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:46.450373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.450351 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:46.452127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.452090 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca" pod "9f37fe2f-7e84-4cf4-9878-51da1dfc8973" (UID: "9f37fe2f-7e84-4cf4-9878-51da1dfc8973"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:46.452348 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.452329 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7" pod "9f37fe2f-7e84-4cf4-9878-51da1dfc8973" (UID: "9f37fe2f-7e84-4cf4-9878-51da1dfc8973"). InnerVolumeSpecName "kube-api-access-68wk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:46.551744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.551706 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wk7\" (UniqueName: \"kubernetes.io/projected/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-kube-api-access-68wk7\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:46.551744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:46.551736 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fe2f-7e84-4cf4-9878-51da1dfc8973-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.228628 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:47.228592 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" event={"ID":"9f37fe2f-7e84-4cf4-9878-51da1dfc8973","Type":"ContainerDied","Data":"090e076d36b43c962a5d44c130ac1fc36e37d08a813a2ef7a222dee7b35bab4f"} Oct 31 00:48:47.229013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:47.228640 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090e076d36b43c962a5d44c130ac1fc36e37d08a813a2ef7a222dee7b35bab4f" Oct 31 00:48:47.229013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:47.228682 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf" Oct 31 00:48:49.221392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.221299 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" podUID="766a21e4-8d3b-46f5-ad52-9514a1834483" containerName="payload-processing" containerID="cri-o://3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1" gracePeriod=25 Oct 31 00:48:49.484279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.484217 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:48:49.587469 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.587439 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm\" (UniqueName: \"kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm\") pod \"766a21e4-8d3b-46f5-ad52-9514a1834483\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " Oct 31 00:48:49.587639 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.587586 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume\") pod \"766a21e4-8d3b-46f5-ad52-9514a1834483\" (UID: \"766a21e4-8d3b-46f5-ad52-9514a1834483\") " Oct 31 00:48:49.587936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.587913 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume" pod "766a21e4-8d3b-46f5-ad52-9514a1834483" (UID: "766a21e4-8d3b-46f5-ad52-9514a1834483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:49.589694 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.589674 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm" pod "766a21e4-8d3b-46f5-ad52-9514a1834483" (UID: "766a21e4-8d3b-46f5-ad52-9514a1834483"). InnerVolumeSpecName "kube-api-access-7x7gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:49.689430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.689397 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a21e4-8d3b-46f5-ad52-9514a1834483-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.689430 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.689424 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x7gm\" (UniqueName: \"kubernetes.io/projected/766a21e4-8d3b-46f5-ad52-9514a1834483-kube-api-access-7x7gm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.847126 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:49.847091 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:50.250618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.250532 5943 generic.go:356] "Generic (PLEG): container finished" podID="766a21e4-8d3b-46f5-ad52-9514a1834483" containerID="3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1" exitCode=0 Oct 31 00:48:50.250618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.250579 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" Oct 31 00:48:50.251108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.250614 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" event={"ID":"766a21e4-8d3b-46f5-ad52-9514a1834483","Type":"ContainerDied","Data":"3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1"} Oct 31 00:48:50.251108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.250678 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5" event={"ID":"766a21e4-8d3b-46f5-ad52-9514a1834483","Type":"ContainerDied","Data":"d843582cf9ece80b62ef168948cdb6891523cf669f0c275b52170f9598541bbb"} Oct 31 00:48:50.251108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.250704 5943 scope.go:122] "RemoveContainer" containerID="3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1" Oct 31 00:48:50.261925 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.261904 5943 scope.go:122] "RemoveContainer" containerID="3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1" Oct 31 00:48:50.262245 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:48:50.262223 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1\": container with ID starting with 3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1 not found: ID does not exist" containerID="3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1" Oct 31 00:48:50.262297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.262258 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1"} err="failed to get container status \"3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1\": rpc error: code = NotFound desc = could not find container \"3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1\": container with ID starting with 3c525ab7253841395c307c1d901750037d244fe9d676d405862447f730a1faa1 not found: ID does not exist" Oct 31 00:48:50.414683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.414646 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5"] Oct 31 00:48:50.418429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:50.418401 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31144e-7c4db79977-cnbx5"] Oct 31 00:48:51.364750 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:51.364713 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.643678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:51.643580 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.795423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:51.795378 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.855637 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:51.855603 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh"] Oct 31 00:48:51.904791 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:51.904711 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ef8827-2b9lh"] Oct 31 00:48:52.042066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:52.042032 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766a21e4-8d3b-46f5-ad52-9514a1834483" path="/var/lib/kubelet/pods/766a21e4-8d3b-46f5-ad52-9514a1834483/volumes" Oct 31 00:48:52.042514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:52.042499 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3" path="/var/lib/kubelet/pods/fee7ab44-286a-4cb6-bf03-7a2ff05eb2e3/volumes" Oct 31 00:48:52.277757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:52.277725 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.427689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:52.427656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.654595 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:52.654498 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.513665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.513632 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn"] Oct 31 00:48:53.514192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.514160 5943 cpu_manager.go:414] "containerMap: removing container" podUID="766a21e4-8d3b-46f5-ad52-9514a1834483" containerName="payload-processing" Oct 31 00:48:53.514192 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.514191 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766a21e4-8d3b-46f5-ad52-9514a1834483" containerName="payload-processing" Oct 31 00:48:53.514280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.514202 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9f37fe2f-7e84-4cf4-9878-51da1dfc8973" containerName="revoke-keys" Oct 31 00:48:53.514280 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.514207 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f37fe2f-7e84-4cf4-9878-51da1dfc8973" containerName="revoke-keys" Oct 31 00:48:53.514347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.514312 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="766a21e4-8d3b-46f5-ad52-9514a1834483" containerName="payload-processing" Oct 31 00:48:53.514347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.514318 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f37fe2f-7e84-4cf4-9878-51da1dfc8973" containerName="revoke-keys" Oct 31 00:48:53.518821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.518805 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.606273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.606233 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.622666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.622639 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:53.622666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.622639 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:53.622666 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.622645 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:53.622951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.622710 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:48:53.627040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.627017 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn"] Oct 31 00:48:53.638950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.638924 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.639069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.638962 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.639111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.639094 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prb22\" (UniqueName: \"kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.740807 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.740760 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.741030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.740823 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.741030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.740914 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prb22\" (UniqueName: \"kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.741644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.741605 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.743370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.743345 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.812737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.812661 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prb22\" (UniqueName: \"kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22\") pod \"maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:53.832274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:53.832245 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:54.217834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:54.217803 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn"] Oct 31 00:48:54.219470 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:48:54.219434 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6514b179_1010_4204_bf9d_20598fba9a02.slice/crio-982bc6071aa48320d35dcb8f6008a4a9075502ceb7af95d7f1124b8f92195d8a WatchSource:0}: Error finding container 982bc6071aa48320d35dcb8f6008a4a9075502ceb7af95d7f1124b8f92195d8a: Status 404 returned error can't find the container with id 982bc6071aa48320d35dcb8f6008a4a9075502ceb7af95d7f1124b8f92195d8a Oct 31 00:48:54.279347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:54.279311 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" event={"ID":"6514b179-1010-4204-bf9d-20598fba9a02","Type":"ContainerStarted","Data":"982bc6071aa48320d35dcb8f6008a4a9075502ceb7af95d7f1124b8f92195d8a"} Oct 31 00:48:55.288050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:55.288016 5943 generic.go:356] "Generic (PLEG): container finished" podID="6514b179-1010-4204-bf9d-20598fba9a02" containerID="84414f09ae8231d366ec1ec40105a587019e818bb8fdf6e8dbb40e9d720926b4" exitCode=0 Oct 31 00:48:55.288546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:55.288066 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" event={"ID":"6514b179-1010-4204-bf9d-20598fba9a02","Type":"ContainerDied","Data":"84414f09ae8231d366ec1ec40105a587019e818bb8fdf6e8dbb40e9d720926b4"} Oct 31 00:48:56.441382 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.441356 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:56.573733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.573634 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca\") pod \"6514b179-1010-4204-bf9d-20598fba9a02\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " Oct 31 00:48:56.573939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.573813 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca\") pod \"6514b179-1010-4204-bf9d-20598fba9a02\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " Oct 31 00:48:56.574025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.573972 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22\" (UniqueName: \"kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22\") pod \"6514b179-1010-4204-bf9d-20598fba9a02\" (UID: \"6514b179-1010-4204-bf9d-20598fba9a02\") " Oct 31 00:48:56.574246 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.574170 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca" pod "6514b179-1010-4204-bf9d-20598fba9a02" (UID: "6514b179-1010-4204-bf9d-20598fba9a02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:56.574512 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.574491 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:56.575875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.575846 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca" pod "6514b179-1010-4204-bf9d-20598fba9a02" (UID: "6514b179-1010-4204-bf9d-20598fba9a02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:56.576157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.576138 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22" pod "6514b179-1010-4204-bf9d-20598fba9a02" (UID: "6514b179-1010-4204-bf9d-20598fba9a02"). InnerVolumeSpecName "kube-api-access-prb22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:56.676120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.676079 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514b179-1010-4204-bf9d-20598fba9a02-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:56.676120 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.676106 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prb22\" (UniqueName: \"kubernetes.io/projected/6514b179-1010-4204-bf9d-20598fba9a02-kube-api-access-prb22\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:56.775685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:56.775641 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:57.303112 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:57.303077 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" Oct 31 00:48:57.303302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:57.303112 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn" event={"ID":"6514b179-1010-4204-bf9d-20598fba9a02","Type":"ContainerDied","Data":"982bc6071aa48320d35dcb8f6008a4a9075502ceb7af95d7f1124b8f92195d8a"} Oct 31 00:48:57.303302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:57.303165 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982bc6071aa48320d35dcb8f6008a4a9075502ceb7af95d7f1124b8f92195d8a" Oct 31 00:48:57.477327 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:48:57.477290 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:48:58.529418 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:58.529384 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:58.678881 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:58.678838 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:58.902816 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:58.902720 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh"] Oct 31 00:48:58.903019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:58.902971 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" containerID="cri-o://7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9" gracePeriod=30 Oct 31 00:48:58.917625 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:48:58.917598 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:01.644867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:01.644833 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:03.321415 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:03.321377 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:49:04.175788 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.175724 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:49:04.263108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263077 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263166 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263250 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263328 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263403 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263530 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263476 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263581 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263536 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263642 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263606 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263702 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263641 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo\") pod \"652b4620-ca12-4f36-910c-057905987c2e\" (UID: \"652b4620-ca12-4f36-910c-057905987c2e\") " Oct 31 00:49:04.263838 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263600 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.263905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263612 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.263905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263685 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:04.263905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263806 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.264071 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.263956 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.264371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.264345 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.264647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.264374 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.264647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.264389 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.264647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.264404 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652b4620-ca12-4f36-910c-057905987c2e-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.264647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.264418 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.265742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.265716 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.265890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.265869 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "kube-api-access-xwxtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:04.266105 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.266079 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:04.266211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.266116 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token" pod "652b4620-ca12-4f36-910c-057905987c2e" (UID: "652b4620-ca12-4f36-910c-057905987c2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:04.355815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.355775 5943 generic.go:356] "Generic (PLEG): container finished" podID="652b4620-ca12-4f36-910c-057905987c2e" containerID="7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9" exitCode=0 Oct 31 00:49:04.356286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.355825 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" Oct 31 00:49:04.356286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.355861 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" event={"ID":"652b4620-ca12-4f36-910c-057905987c2e","Type":"ContainerDied","Data":"7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9"} Oct 31 00:49:04.356286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.355939 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh" event={"ID":"652b4620-ca12-4f36-910c-057905987c2e","Type":"ContainerDied","Data":"52b6655555e332dedeb056c493bb495ae2b262e08f9c7dfdda32ac352f79fefc"} Oct 31 00:49:04.356286 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.355964 5943 scope.go:122] "RemoveContainer" containerID="7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9" Oct 31 00:49:04.365938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.365914 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652b4620-ca12-4f36-910c-057905987c2e-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.366047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.365938 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.366047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.365953 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxtz\" (UniqueName: \"kubernetes.io/projected/652b4620-ca12-4f36-910c-057905987c2e-kube-api-access-xwxtz\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.366047 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.365967 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652b4620-ca12-4f36-910c-057905987c2e-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.367220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.367195 5943 scope.go:122] "RemoveContainer" containerID="aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d" Oct 31 00:49:04.376801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.376782 5943 scope.go:122] "RemoveContainer" containerID="7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9" Oct 31 00:49:04.377132 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:49:04.377110 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9\": container with ID starting with 7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9 not found: ID does not exist" containerID="7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9" Oct 31 00:49:04.377327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.377136 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9"} err="failed to get container status \"7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9\": rpc error: code = NotFound desc = could not find container \"7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9\": container with ID starting with 7606424ece941f77ced66075277df40eea19b6db022422759e89efaeb18166b9 not found: ID does not exist" Oct 31 00:49:04.377327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.377154 5943 scope.go:122] "RemoveContainer" containerID="aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d" Oct 31 00:49:04.377429 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:49:04.377404 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d\": container with ID starting with aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d not found: ID does not exist" containerID="aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d" Oct 31 00:49:04.377429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.377423 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d"} err="failed to get container status \"aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d\": rpc error: code = NotFound desc = could not find container \"aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d\": container with ID starting with aed02e94d57771f5073351bbf2d14a540fbcb0f8f20b6c3c3fbee6049188a58d not found: ID does not exist" Oct 31 00:49:04.536668 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.536034 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh"] Oct 31 00:49:04.541966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.541935 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ef8827-openshift-default-6f86df96f-rlggh"] Oct 31 00:49:04.600316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:04.600269 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.211485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:05.211455 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p"] Oct 31 00:49:05.468942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:05.468845 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.614822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:05.614782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:06.041450 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.041414 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652b4620-ca12-4f36-910c-057905987c2e" path="/var/lib/kubelet/pods/652b4620-ca12-4f36-910c-057905987c2e/volumes" Oct 31 00:49:06.461849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.461711 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:06.866388 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.866346 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b"] Oct 31 00:49:06.867080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867059 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6514b179-1010-4204-bf9d-20598fba9a02" containerName="revoke-keys" Oct 31 00:49:06.867205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867081 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6514b179-1010-4204-bf9d-20598fba9a02" containerName="revoke-keys" Oct 31 00:49:06.867205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867102 5943 cpu_manager.go:414] "containerMap: removing container" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" Oct 31 00:49:06.867205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867109 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" Oct 31 00:49:06.867205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867133 5943 cpu_manager.go:414] "containerMap: removing container" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" Oct 31 00:49:06.867205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867141 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" Oct 31 00:49:06.867455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867293 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" Oct 31 00:49:06.867455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867311 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="652b4620-ca12-4f36-910c-057905987c2e" containerName="istio-proxy" Oct 31 00:49:06.867455 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.867335 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6514b179-1010-4204-bf9d-20598fba9a02" containerName="revoke-keys" Oct 31 00:49:06.870997 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.870973 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:06.916963 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.916928 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:06.917134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.917054 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnrs\" (UniqueName: \"kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:06.917134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.917127 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:06.925965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.925934 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:06.929167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.929144 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b"] Oct 31 00:49:06.974971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.974939 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:06.974971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.974939 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:06.975216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:06.974996 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:49:07.017980 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.017946 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.018236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.018029 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.018236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.018091 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnrs\" (UniqueName: \"kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.018744 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.018724 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.020679 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.020656 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.078991 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.078949 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnrs\" (UniqueName: \"kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs\") pod \"maas-api-revoke-sub-96b3c7a1-n5g2b\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.184830 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.184738 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:07.577250 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:07.571053 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b"] Oct 31 00:49:07.579172 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:49:07.579141 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e868180_588c_4084_85c5_9f6d4ac9c3ee.slice/crio-e3a97db5dbf6aba3cab6c11e210d69f10529e6b1e980227c30832457c76ef80b WatchSource:0}: Error finding container e3a97db5dbf6aba3cab6c11e210d69f10529e6b1e980227c30832457c76ef80b: Status 404 returned error can't find the container with id e3a97db5dbf6aba3cab6c11e210d69f10529e6b1e980227c30832457c76ef80b Oct 31 00:49:08.384537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:08.384498 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" event={"ID":"6e868180-588c-4084-85c5-9f6d4ac9c3ee","Type":"ContainerStarted","Data":"2cb87f8ec309574f051dacc3b94ee8efc066346ac8c3801caa9503a2e62122fa"} Oct 31 00:49:08.384952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:08.384553 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" event={"ID":"6e868180-588c-4084-85c5-9f6d4ac9c3ee","Type":"ContainerStarted","Data":"e3a97db5dbf6aba3cab6c11e210d69f10529e6b1e980227c30832457c76ef80b"} Oct 31 00:49:08.500995 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:08.500941 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" podStartSLOduration=2.500921222 podStartE2EDuration="2.500921222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:08.498957855 +0000 UTC m=+2728.592248043" watchObservedRunningTime="2026-10-31 00:49:08.500921222 +0000 UTC m=+2728.594211410" Oct 31 00:49:09.392718 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:09.392681 5943 generic.go:356] "Generic (PLEG): container finished" podID="6e868180-588c-4084-85c5-9f6d4ac9c3ee" containerID="2cb87f8ec309574f051dacc3b94ee8efc066346ac8c3801caa9503a2e62122fa" exitCode=0 Oct 31 00:49:09.393127 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:09.392726 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" event={"ID":"6e868180-588c-4084-85c5-9f6d4ac9c3ee","Type":"ContainerDied","Data":"2cb87f8ec309574f051dacc3b94ee8efc066346ac8c3801caa9503a2e62122fa"} Oct 31 00:49:10.212362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.212319 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" podUID="db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" containerName="payload-pre-processing" containerID="cri-o://f85481c6db8f5a7658c542598e574d9e5eff70e4cb672a4730effddfcce71568" gracePeriod=25 Oct 31 00:49:10.405209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.405142 5943 generic.go:356] "Generic (PLEG): container finished" podID="db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" containerID="f85481c6db8f5a7658c542598e574d9e5eff70e4cb672a4730effddfcce71568" exitCode=0 Oct 31 00:49:10.405691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.405229 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" event={"ID":"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1","Type":"ContainerDied","Data":"f85481c6db8f5a7658c542598e574d9e5eff70e4cb672a4730effddfcce71568"} Oct 31 00:49:10.485972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.485940 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:49:10.557162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.557138 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:10.558526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.558505 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume\") pod \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " Oct 31 00:49:10.558633 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.558552 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5\" (UniqueName: \"kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5\") pod \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\" (UID: \"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1\") " Oct 31 00:49:10.558842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.558822 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume" pod "db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" (UID: "db3fd683-beb8-4d2e-a4d7-e66aaaff4df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:10.559005 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.558989 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:10.560697 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.560676 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5" pod "db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" (UID: "db3fd683-beb8-4d2e-a4d7-e66aaaff4df1"). InnerVolumeSpecName "kube-api-access-pn2w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:10.660110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.660081 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs\" (UniqueName: \"kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs\") pod \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " Oct 31 00:49:10.660312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.660157 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca\") pod \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " Oct 31 00:49:10.660312 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.660206 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca\") pod \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\" (UID: \"6e868180-588c-4084-85c5-9f6d4ac9c3ee\") " Oct 31 00:49:10.660675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.660641 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca" pod "6e868180-588c-4084-85c5-9f6d4ac9c3ee" (UID: "6e868180-588c-4084-85c5-9f6d4ac9c3ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:10.660803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.660651 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2w5\" (UniqueName: \"kubernetes.io/projected/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1-kube-api-access-pn2w5\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:10.662572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.662542 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs" pod "6e868180-588c-4084-85c5-9f6d4ac9c3ee" (UID: "6e868180-588c-4084-85c5-9f6d4ac9c3ee"). InnerVolumeSpecName "kube-api-access-kfnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:10.662665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.662642 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca" pod "6e868180-588c-4084-85c5-9f6d4ac9c3ee" (UID: "6e868180-588c-4084-85c5-9f6d4ac9c3ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:10.761641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.761611 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfnrs\" (UniqueName: \"kubernetes.io/projected/6e868180-588c-4084-85c5-9f6d4ac9c3ee-kube-api-access-kfnrs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:10.761782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.761647 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:10.761782 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:10.761657 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e868180-588c-4084-85c5-9f6d4ac9c3ee-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:11.251597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.251558 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.404165 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.404129 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.412846 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.412821 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" Oct 31 00:49:11.413244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.412848 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b" event={"ID":"6e868180-588c-4084-85c5-9f6d4ac9c3ee","Type":"ContainerDied","Data":"e3a97db5dbf6aba3cab6c11e210d69f10529e6b1e980227c30832457c76ef80b"} Oct 31 00:49:11.413244 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.412896 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a97db5dbf6aba3cab6c11e210d69f10529e6b1e980227c30832457c76ef80b" Oct 31 00:49:11.414597 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.414570 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" Oct 31 00:49:11.414672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.414573 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p" event={"ID":"db3fd683-beb8-4d2e-a4d7-e66aaaff4df1","Type":"ContainerDied","Data":"3bc7b90ffbaafc0e1d19ee2c77426cd58d7f07873debed9ef399cb80102fee78"} Oct 31 00:49:11.414713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.414690 5943 scope.go:122] "RemoveContainer" containerID="f85481c6db8f5a7658c542598e574d9e5eff70e4cb672a4730effddfcce71568" Oct 31 00:49:11.581011 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.580969 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p"] Oct 31 00:49:11.593534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.593498 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bf38e4-77fff9db6c-lrg2p"] Oct 31 00:49:11.633453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.633422 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.880611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:11.880503 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:12.027091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:12.027056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:12.040710 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:12.040678 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" path="/var/lib/kubelet/pods/db3fd683-beb8-4d2e-a4d7-e66aaaff4df1/volumes" Oct 31 00:49:16.232985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:16.232943 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.805328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.805291 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s"] Oct 31 00:49:17.806129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.806104 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6e868180-588c-4084-85c5-9f6d4ac9c3ee" containerName="revoke-keys" Oct 31 00:49:17.806225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.806133 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e868180-588c-4084-85c5-9f6d4ac9c3ee" containerName="revoke-keys" Oct 31 00:49:17.806225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.806167 5943 cpu_manager.go:414] "containerMap: removing container" podUID="db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" containerName="payload-pre-processing" Oct 31 00:49:17.806225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.806193 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" containerName="payload-pre-processing" Oct 31 00:49:17.806378 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.806364 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e868180-588c-4084-85c5-9f6d4ac9c3ee" containerName="revoke-keys" Oct 31 00:49:17.806412 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.806386 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3fd683-beb8-4d2e-a4d7-e66aaaff4df1" containerName="payload-pre-processing" Oct 31 00:49:17.810001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.809986 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:17.866712 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.866681 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s"] Oct 31 00:49:17.913241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.913168 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:17.913439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.913342 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:17.913439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.913364 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:49:17.913439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.913369 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:17.939008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.938965 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:17.939148 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.939088 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:17.939232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:17.939162 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znws4\" (UniqueName: \"kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.040634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.040602 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.040842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.040663 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znws4\" (UniqueName: \"kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.040842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.040749 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.041445 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.041374 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.043157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.043135 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.102361 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.102284 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znws4\" (UniqueName: \"kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4\") pod \"maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.124023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.123996 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:18.501279 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:18.501250 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s"] Oct 31 00:49:18.502513 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:49:18.502485 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87d2f71_3a56_4980_80a6_7ecdaf705aea.slice/crio-29e0f0154240a78f91c26d2de0ffd792d6e26cce47d74bc7180d7f98f0247f2f WatchSource:0}: Error finding container 29e0f0154240a78f91c26d2de0ffd792d6e26cce47d74bc7180d7f98f0247f2f: Status 404 returned error can't find the container with id 29e0f0154240a78f91c26d2de0ffd792d6e26cce47d74bc7180d7f98f0247f2f Oct 31 00:49:19.479103 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:19.479064 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" event={"ID":"b87d2f71-3a56-4980-80a6-7ecdaf705aea","Type":"ContainerStarted","Data":"d1d72b746a21fd00d9159902840fd7dc57512e523a1b573e33032ee65fd16c23"} Oct 31 00:49:19.479568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:19.479125 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" event={"ID":"b87d2f71-3a56-4980-80a6-7ecdaf705aea","Type":"ContainerStarted","Data":"29e0f0154240a78f91c26d2de0ffd792d6e26cce47d74bc7180d7f98f0247f2f"} Oct 31 00:49:19.593738 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:19.593687 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" podStartSLOduration=2.593674063 podStartE2EDuration="2.593674063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:19.593452337 +0000 UTC m=+2739.686742536" watchObservedRunningTime="2026-10-31 00:49:19.593674063 +0000 UTC m=+2739.686964250" Oct 31 00:49:20.488251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:20.488217 5943 generic.go:356] "Generic (PLEG): container finished" podID="b87d2f71-3a56-4980-80a6-7ecdaf705aea" containerID="d1d72b746a21fd00d9159902840fd7dc57512e523a1b573e33032ee65fd16c23" exitCode=0 Oct 31 00:49:20.488251 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:20.488252 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" event={"ID":"b87d2f71-3a56-4980-80a6-7ecdaf705aea","Type":"ContainerDied","Data":"d1d72b746a21fd00d9159902840fd7dc57512e523a1b573e33032ee65fd16c23"} Oct 31 00:49:20.719236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:20.719199 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:21.669235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.669212 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:21.783048 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.783008 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4\" (UniqueName: \"kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4\") pod \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " Oct 31 00:49:21.783269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.783062 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca\") pod \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " Oct 31 00:49:21.783269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.783098 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca\") pod \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\" (UID: \"b87d2f71-3a56-4980-80a6-7ecdaf705aea\") " Oct 31 00:49:21.783513 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.783477 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca" pod "b87d2f71-3a56-4980-80a6-7ecdaf705aea" (UID: "b87d2f71-3a56-4980-80a6-7ecdaf705aea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:21.783721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.783691 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.785686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.785660 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4" pod "b87d2f71-3a56-4980-80a6-7ecdaf705aea" (UID: "b87d2f71-3a56-4980-80a6-7ecdaf705aea"). InnerVolumeSpecName "kube-api-access-znws4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:21.785870 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.785852 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca" pod "b87d2f71-3a56-4980-80a6-7ecdaf705aea" (UID: "b87d2f71-3a56-4980-80a6-7ecdaf705aea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:21.885124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.885086 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87d2f71-3a56-4980-80a6-7ecdaf705aea-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.885124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:21.885125 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znws4\" (UniqueName: \"kubernetes.io/projected/b87d2f71-3a56-4980-80a6-7ecdaf705aea-kube-api-access-znws4\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:22.248950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.248913 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:22.467612 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.467520 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:22.507732 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.507706 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" Oct 31 00:49:22.507904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.507745 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s" event={"ID":"b87d2f71-3a56-4980-80a6-7ecdaf705aea","Type":"ContainerDied","Data":"29e0f0154240a78f91c26d2de0ffd792d6e26cce47d74bc7180d7f98f0247f2f"} Oct 31 00:49:22.507904 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.507792 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e0f0154240a78f91c26d2de0ffd792d6e26cce47d74bc7180d7f98f0247f2f" Oct 31 00:49:22.610269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.610233 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:22.751452 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.751365 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq"] Oct 31 00:49:22.754854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:22.754826 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-31144e-78wvq"] Oct 31 00:49:23.205585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:23.205550 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:24.042373 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:24.042333 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eea6a98-acca-49cd-9d34-06490c9c1670" path="/var/lib/kubelet/pods/6eea6a98-acca-49cd-9d34-06490c9c1670/volumes" Oct 31 00:49:27.448798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:27.448751 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:27.475981 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:49:27.475940 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:49:29.450678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:29.450635 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:29.519888 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:29.519849 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q"] Oct 31 00:49:29.652809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:29.652768 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:29.831240 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:29.831209 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:29.985098 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:29.985064 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:31.002909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:31.002872 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:49:31.716701 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:31.716660 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.259722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:33.259685 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.489896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:33.489861 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.639724 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:33.639629 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.775803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:33.775770 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn"] Oct 31 00:49:33.824743 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:33.824712 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bf38e4-2tvbn"] Oct 31 00:49:34.043200 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.041706 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6514b179-1010-4204-bf9d-20598fba9a02" path="/var/lib/kubelet/pods/6514b179-1010-4204-bf9d-20598fba9a02/volumes" Oct 31 00:49:34.263783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.263746 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:34.520298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.520259 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" podUID="4c74d794-8738-4f7e-b3b8-cbc07f6f48df" containerName="payload-pre-processing" containerID="cri-o://266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03" gracePeriod=25 Oct 31 00:49:34.776779 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.776707 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:49:34.839843 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.839804 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume\") pod \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " Oct 31 00:49:34.840043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.840026 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8\" (UniqueName: \"kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8\") pod \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\" (UID: \"4c74d794-8738-4f7e-b3b8-cbc07f6f48df\") " Oct 31 00:49:34.840211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.840159 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume" pod "4c74d794-8738-4f7e-b3b8-cbc07f6f48df" (UID: "4c74d794-8738-4f7e-b3b8-cbc07f6f48df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:34.840515 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.840493 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:34.842385 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.842359 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8" pod "4c74d794-8738-4f7e-b3b8-cbc07f6f48df" (UID: "4c74d794-8738-4f7e-b3b8-cbc07f6f48df"). InnerVolumeSpecName "kube-api-access-cl2h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:34.942389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:34.942355 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl2h8\" (UniqueName: \"kubernetes.io/projected/4c74d794-8738-4f7e-b3b8-cbc07f6f48df-kube-api-access-cl2h8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:35.604704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.604664 5943 generic.go:356] "Generic (PLEG): container finished" podID="4c74d794-8738-4f7e-b3b8-cbc07f6f48df" containerID="266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03" exitCode=0 Oct 31 00:49:35.605152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.604712 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" event={"ID":"4c74d794-8738-4f7e-b3b8-cbc07f6f48df","Type":"ContainerDied","Data":"266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03"} Oct 31 00:49:35.605152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.604722 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" Oct 31 00:49:35.605152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.604773 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q" event={"ID":"4c74d794-8738-4f7e-b3b8-cbc07f6f48df","Type":"ContainerDied","Data":"f4ee4c03ed0a28896d21069abe26c732aabce20ef4654e034d5973b9b7a3dcfb"} Oct 31 00:49:35.605152 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.604804 5943 scope.go:122] "RemoveContainer" containerID="266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03" Oct 31 00:49:35.615704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.615687 5943 scope.go:122] "RemoveContainer" containerID="266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03" Oct 31 00:49:35.616104 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:49:35.616078 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03\": container with ID starting with 266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03 not found: ID does not exist" containerID="266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03" Oct 31 00:49:35.616228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.616117 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03"} err="failed to get container status \"266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03\": rpc error: code = NotFound desc = could not find container \"266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03\": container with ID starting with 266c779ea79f0ac8531af8d471642e1bfdb961f14e232dda23d990b194d23c03 not found: ID does not exist" Oct 31 00:49:35.770767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.770738 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q"] Oct 31 00:49:35.801486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:35.801453 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-868b06-649dbfcd47-vwl6q"] Oct 31 00:49:36.041377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:36.041343 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c74d794-8738-4f7e-b3b8-cbc07f6f48df" path="/var/lib/kubelet/pods/4c74d794-8738-4f7e-b3b8-cbc07f6f48df/volumes" Oct 31 00:49:38.655269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:38.655232 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.363108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:40.363070 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.561058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:40.561013 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.744620 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:40.744539 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.047402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.047363 5943 scope.go:122] "RemoveContainer" containerID="e4b77e4bd800c4638ec278dc510674fa44b3776117f537148d75ba12af3e3819" Oct 31 00:49:41.057345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.057322 5943 scope.go:122] "RemoveContainer" containerID="44ffd1de48bf5b985767278683298abc534da4827e3825d93e5e95c0bca021b7" Oct 31 00:49:41.066785 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.066768 5943 scope.go:122] "RemoveContainer" containerID="8f79dba6316f825719d6887de869e1a22c4713138169ee1cb04011499aa5d3af" Oct 31 00:49:41.796752 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.796719 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx"] Oct 31 00:49:41.797245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.797230 5943 cpu_manager.go:414] "containerMap: removing container" podUID="4c74d794-8738-4f7e-b3b8-cbc07f6f48df" containerName="payload-pre-processing" Oct 31 00:49:41.797300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.797245 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c74d794-8738-4f7e-b3b8-cbc07f6f48df" containerName="payload-pre-processing" Oct 31 00:49:41.797300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.797257 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b87d2f71-3a56-4980-80a6-7ecdaf705aea" containerName="revoke-keys" Oct 31 00:49:41.797300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.797262 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87d2f71-3a56-4980-80a6-7ecdaf705aea" containerName="revoke-keys" Oct 31 00:49:41.797397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.797358 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c74d794-8738-4f7e-b3b8-cbc07f6f48df" containerName="payload-pre-processing" Oct 31 00:49:41.797397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.797366 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87d2f71-3a56-4980-80a6-7ecdaf705aea" containerName="revoke-keys" Oct 31 00:49:41.802160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.802139 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.891340 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.891307 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx"] Oct 31 00:49:41.936596 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.936558 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.936799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.936614 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.936799 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.936726 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.936937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.936848 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.936937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.936904 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.937025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.936957 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.937063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.937019 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mj5h\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.937100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.937063 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:41.937143 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:41.937102 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.037923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.037886 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.037923 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.037927 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.037959 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038022 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038058 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038211 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038103 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038309 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mj5h\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038361 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038365 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038383 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038442 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038508 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038438 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.038996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.038974 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.040754 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.040732 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.041108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.041086 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.098996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.098923 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.098996 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.098972 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mj5h\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.115795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.115768 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:42.548134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.548095 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx"] Oct 31 00:49:42.550872 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:49:42.550841 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658a4219_b591_4867_8511_86823dfd9f14.slice/crio-90494068bde875403ab306fb94ba4b61d293b6bebe930bdcc3a0e69b49c72c3e WatchSource:0}: Error finding container 90494068bde875403ab306fb94ba4b61d293b6bebe930bdcc3a0e69b49c72c3e: Status 404 returned error can't find the container with id 90494068bde875403ab306fb94ba4b61d293b6bebe930bdcc3a0e69b49c72c3e Oct 31 00:49:42.553653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.553618 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:49:42.553759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.553703 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:49:42.553759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.553746 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:49:42.658957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.658913 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" event={"ID":"658a4219-b591-4867-8511-86823dfd9f14","Type":"ContainerStarted","Data":"535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25"} Oct 31 00:49:42.659137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.658973 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" event={"ID":"658a4219-b591-4867-8511-86823dfd9f14","Type":"ContainerStarted","Data":"90494068bde875403ab306fb94ba4b61d293b6bebe930bdcc3a0e69b49c72c3e"} Oct 31 00:49:42.780753 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.780699 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podStartSLOduration=1.7806845770000002 podStartE2EDuration="1.780684577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:42.778569436 +0000 UTC m=+2762.871859682" watchObservedRunningTime="2026-10-31 00:49:42.780684577 +0000 UTC m=+2762.873974765" Oct 31 00:49:42.965534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:42.965449 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:43.116935 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:43.116896 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:43.118661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:43.118634 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:43.118803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:43.118682 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:43.451276 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:43.451240 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:43.855893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:43.855813 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.011798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:44.011764 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.116912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:44.116829 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:44.116912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:44.116884 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:45.116650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:45.116598 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:45.117150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:45.116707 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:46.116645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:46.116611 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:46.116817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:46.116668 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:46.955018 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:46.954977 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:47.116646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:47.116607 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:47.116832 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:47.116676 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:47.719546 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:47.719517 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:47.952589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:47.952498 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.117441 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:48.117408 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:48.118006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:48.117973 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:48.181257 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:48.181217 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.354198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:48.354142 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.528563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:48.528528 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:49.116939 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:49.116895 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:49:49.117160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:49.116971 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:49:49.863977 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:49.863927 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.121345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.121228 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:50.121674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.121635 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:50.122518 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.122487 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:49:50.199866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.199826 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.404062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.403983 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.421862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.421824 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s"] Oct 31 00:49:50.434269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.434240 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-868b06-sq26s"] Oct 31 00:49:50.922514 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:50.922477 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:52.041008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:52.040970 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87d2f71-3a56-4980-80a6-7ecdaf705aea" path="/var/lib/kubelet/pods/b87d2f71-3a56-4980-80a6-7ecdaf705aea/volumes" Oct 31 00:49:52.482568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:52.482475 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:52.688298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:52.688263 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:53.179462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:53.179427 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:53.476080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:53.475991 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:53.637130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:53.637090 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:53.974711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:53.974673 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:55.499808 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:55.499777 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.017603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.017565 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.182193 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.182148 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.360965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.360895 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp"] Oct 31 00:49:57.361167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.361146 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" containerID="cri-o://ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8" gracePeriod=30 Oct 31 00:49:57.368355 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.368323 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.482917 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:49:57.482879 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:49:57.625298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.625217 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.811545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:57.811511 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:58.611131 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:58.611090 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:49:59.770895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:49:59.770852 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:00.515417 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:00.515383 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:00.903866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:00.903769 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:01.122841 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:01.122808 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:02.633325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.633265 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:50:02.792524 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792485 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792581 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792618 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792671 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792726 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792707 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792735 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792831 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.792950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792866 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.793094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792950 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs\") pod \"ea6edfaa-bdcc-4657-9055-89773e6056ae\" (UID: \"ea6edfaa-bdcc-4657-9055-89773e6056ae\") " Oct 31 00:50:02.793094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.792983 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:02.793094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793005 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:02.793094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793051 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:02.793329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793307 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:02.793461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793444 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.793532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793473 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.793532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793489 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.793532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793503 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.793654 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.793549 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:02.794989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.794968 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:02.795337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.795317 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "kube-api-access-vtll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:02.795486 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.795470 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:02.795550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.795526 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo" pod "ea6edfaa-bdcc-4657-9055-89773e6056ae" (UID: "ea6edfaa-bdcc-4657-9055-89773e6056ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:02.821428 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.821395 5943 generic.go:356] "Generic (PLEG): container finished" podID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerID="ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8" exitCode=0 Oct 31 00:50:02.821574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.821452 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" Oct 31 00:50:02.821574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.821478 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" event={"ID":"ea6edfaa-bdcc-4657-9055-89773e6056ae","Type":"ContainerDied","Data":"ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8"} Oct 31 00:50:02.821574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.821535 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp" event={"ID":"ea6edfaa-bdcc-4657-9055-89773e6056ae","Type":"ContainerDied","Data":"0cff27dbdd6cba6b5322926b165c45e7e497861618069467d2007b890f7e5b8f"} Oct 31 00:50:02.821574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.821556 5943 scope.go:122] "RemoveContainer" containerID="ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8" Oct 31 00:50:02.834249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.834223 5943 scope.go:122] "RemoveContainer" containerID="015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e" Oct 31 00:50:02.845124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.845100 5943 scope.go:122] "RemoveContainer" containerID="ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8" Oct 31 00:50:02.845439 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:50:02.845420 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8\": container with ID starting with ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8 not found: ID does not exist" containerID="ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8" Oct 31 00:50:02.845533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.845447 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8"} err="failed to get container status \"ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8\": rpc error: code = NotFound desc = could not find container \"ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8\": container with ID starting with ca7a4aa80432429f53cddf435cf4031934c79391511861009db14b3ba61e41a8 not found: ID does not exist" Oct 31 00:50:02.845533 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.845466 5943 scope.go:122] "RemoveContainer" containerID="015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e" Oct 31 00:50:02.845724 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:50:02.845710 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e\": container with ID starting with 015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e not found: ID does not exist" containerID="015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e" Oct 31 00:50:02.845770 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.845729 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e"} err="failed to get container status \"015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e\": rpc error: code = NotFound desc = could not find container \"015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e\": container with ID starting with 015e7b371482b3ce96f3a5e5532bd24cdeaf13b9ed8969e1528d4653b423464e not found: ID does not exist" Oct 31 00:50:02.880914 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.880880 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28"] Oct 31 00:50:02.881503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.881481 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" Oct 31 00:50:02.881503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.881500 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" Oct 31 00:50:02.881660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.881523 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" Oct 31 00:50:02.881660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.881528 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" Oct 31 00:50:02.881660 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.881613 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" Oct 31 00:50:02.886856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.886794 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:02.894258 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.894233 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.894371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.894262 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-kube-api-access-vtll9\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.894371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.894280 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.894371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.894293 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6edfaa-bdcc-4657-9055-89773e6056ae-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.894371 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.894307 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6edfaa-bdcc-4657-9055-89773e6056ae-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:02.940644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.940613 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:50:02.940851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.940613 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:02.940851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.940645 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:02.940851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.940764 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:02.944823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.944637 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28"] Oct 31 00:50:02.988898 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.988863 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp"] Oct 31 00:50:02.994810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.994783 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-868b06-openshift-default-58984b7645-274sp"] Oct 31 00:50:02.994986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.994888 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbzfk\" (UniqueName: \"kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:02.995084 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.995066 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:02.995157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:02.995094 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.096670 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.096640 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbzfk\" (UniqueName: \"kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.096880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.096752 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.096880 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.096774 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.097448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.097413 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.099495 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.099473 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.157908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.157826 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbzfk\" (UniqueName: \"kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk\") pod \"maas-api-revoke-sub-269cbaab-twv28\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.199805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.199778 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:03.575524 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.575492 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28"] Oct 31 00:50:03.576677 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:50:03.576648 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8482d258_1d47_4a7b_beb0_bc274f3e302c.slice/crio-db13c76b8bffeed77ad84af4df87ffcd5080b1e0788eced51dea68d88bb31fd3 WatchSource:0}: Error finding container db13c76b8bffeed77ad84af4df87ffcd5080b1e0788eced51dea68d88bb31fd3: Status 404 returned error can't find the container with id db13c76b8bffeed77ad84af4df87ffcd5080b1e0788eced51dea68d88bb31fd3 Oct 31 00:50:03.829822 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:03.829732 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" event={"ID":"8482d258-1d47-4a7b-beb0-bc274f3e302c","Type":"ContainerStarted","Data":"db13c76b8bffeed77ad84af4df87ffcd5080b1e0788eced51dea68d88bb31fd3"} Oct 31 00:50:04.041254 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:04.041217 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" path="/var/lib/kubelet/pods/ea6edfaa-bdcc-4657-9055-89773e6056ae/volumes" Oct 31 00:50:04.839233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:04.839169 5943 generic.go:356] "Generic (PLEG): container finished" podID="8482d258-1d47-4a7b-beb0-bc274f3e302c" containerID="34aab1ccce13a84beb0388e4a983ef93dd9387ee3bc331563e0538354a459ff8" exitCode=0 Oct 31 00:50:04.839233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:04.839210 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" event={"ID":"8482d258-1d47-4a7b-beb0-bc274f3e302c","Type":"ContainerDied","Data":"34aab1ccce13a84beb0388e4a983ef93dd9387ee3bc331563e0538354a459ff8"} Oct 31 00:50:05.139792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:05.139702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.323406 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:05.323350 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.508169 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:05.508123 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.995379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:05.995356 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:06.042360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.042325 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:06.133768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.133677 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca\") pod \"8482d258-1d47-4a7b-beb0-bc274f3e302c\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " Oct 31 00:50:06.133938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.133766 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca\") pod \"8482d258-1d47-4a7b-beb0-bc274f3e302c\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " Oct 31 00:50:06.133938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.133801 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk\" (UniqueName: \"kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk\") pod \"8482d258-1d47-4a7b-beb0-bc274f3e302c\" (UID: \"8482d258-1d47-4a7b-beb0-bc274f3e302c\") " Oct 31 00:50:06.134075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.134051 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca" pod "8482d258-1d47-4a7b-beb0-bc274f3e302c" (UID: "8482d258-1d47-4a7b-beb0-bc274f3e302c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:06.134540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.134513 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.136216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.136171 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca" pod "8482d258-1d47-4a7b-beb0-bc274f3e302c" (UID: "8482d258-1d47-4a7b-beb0-bc274f3e302c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:06.136314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.136255 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk" pod "8482d258-1d47-4a7b-beb0-bc274f3e302c" (UID: "8482d258-1d47-4a7b-beb0-bc274f3e302c"). InnerVolumeSpecName "kube-api-access-mbzfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:06.235537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.235503 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8482d258-1d47-4a7b-beb0-bc274f3e302c-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.235537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.235533 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbzfk\" (UniqueName: \"kubernetes.io/projected/8482d258-1d47-4a7b-beb0-bc274f3e302c-kube-api-access-mbzfk\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.853602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.853568 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" Oct 31 00:50:06.853783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.853596 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28" event={"ID":"8482d258-1d47-4a7b-beb0-bc274f3e302c","Type":"ContainerDied","Data":"db13c76b8bffeed77ad84af4df87ffcd5080b1e0788eced51dea68d88bb31fd3"} Oct 31 00:50:06.853783 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:06.853647 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db13c76b8bffeed77ad84af4df87ffcd5080b1e0788eced51dea68d88bb31fd3" Oct 31 00:50:08.940722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:08.940679 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.087598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:09.087557 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.559992 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:09.559953 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.735834 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:09.735801 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:10.953534 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:10.953501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:15.666212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:15.666160 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.396890 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:20.396809 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.584069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:20.584036 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.734957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:20.734860 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:23.577999 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:23.577960 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:50:25.422571 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:25.422539 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.594921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:25.594875 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.781972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:25.781938 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.040930 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.040846 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55"] Oct 31 00:50:26.041489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.041468 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8482d258-1d47-4a7b-beb0-bc274f3e302c" containerName="revoke-keys" Oct 31 00:50:26.041489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.041489 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8482d258-1d47-4a7b-beb0-bc274f3e302c" containerName="revoke-keys" Oct 31 00:50:26.041673 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.041602 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8482d258-1d47-4a7b-beb0-bc274f3e302c" containerName="revoke-keys" Oct 31 00:50:26.041673 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.041625 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6edfaa-bdcc-4657-9055-89773e6056ae" containerName="istio-proxy" Oct 31 00:50:26.045911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.045890 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.096216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.096144 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55"] Oct 31 00:50:26.108480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.108446 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwrw\" (UniqueName: \"kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.108713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.108527 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.108713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.108671 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.142074 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.142038 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:26.142263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.142073 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:26.142263 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.142080 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:26.147449 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.147427 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:50:26.158590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.158562 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.209555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.209524 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.209737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.209573 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.209737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.209703 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwrw\" (UniqueName: \"kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.210241 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.210218 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.212130 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.212103 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.270664 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.270629 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwrw\" (UniqueName: \"kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw\") pod \"maas-api-revoke-sub-78b86695-6rx55\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.363102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.363030 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:26.744437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.744403 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55"] Oct 31 00:50:26.916840 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:26.916789 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.007708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:27.007678 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" event={"ID":"5bfb90df-62a9-4acd-afc1-3df560652008","Type":"ContainerStarted","Data":"20d7fa71533762fa625c3116c0efbe2f15d7373d068b65cfd9a64f5823c182f1"} Oct 31 00:50:27.138551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:27.138514 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.482748 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:50:27.482639 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:50:28.017403 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:28.017365 5943 generic.go:356] "Generic (PLEG): container finished" podID="5bfb90df-62a9-4acd-afc1-3df560652008" containerID="3de1410030f72bea15eac2432a72905c70b88ed10c4acc299744b72c42a19b1f" exitCode=0 Oct 31 00:50:28.017864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:28.017441 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" event={"ID":"5bfb90df-62a9-4acd-afc1-3df560652008","Type":"ContainerDied","Data":"3de1410030f72bea15eac2432a72905c70b88ed10c4acc299744b72c42a19b1f"} Oct 31 00:50:29.178515 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.178479 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:29.247021 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.246989 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca\") pod \"5bfb90df-62a9-4acd-afc1-3df560652008\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " Oct 31 00:50:29.247245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.247121 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca\") pod \"5bfb90df-62a9-4acd-afc1-3df560652008\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " Oct 31 00:50:29.247338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.247280 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw\" (UniqueName: \"kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw\") pod \"5bfb90df-62a9-4acd-afc1-3df560652008\" (UID: \"5bfb90df-62a9-4acd-afc1-3df560652008\") " Oct 31 00:50:29.247435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.247412 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca" pod "5bfb90df-62a9-4acd-afc1-3df560652008" (UID: "5bfb90df-62a9-4acd-afc1-3df560652008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:29.247707 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.247683 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:29.249453 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.249426 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca" pod "5bfb90df-62a9-4acd-afc1-3df560652008" (UID: "5bfb90df-62a9-4acd-afc1-3df560652008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:29.249556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.249489 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw" pod "5bfb90df-62a9-4acd-afc1-3df560652008" (UID: "5bfb90df-62a9-4acd-afc1-3df560652008"). InnerVolumeSpecName "kube-api-access-2zwrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:29.348517 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.348423 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb90df-62a9-4acd-afc1-3df560652008-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:29.348517 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:29.348453 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zwrw\" (UniqueName: \"kubernetes.io/projected/5bfb90df-62a9-4acd-afc1-3df560652008-kube-api-access-2zwrw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:30.031106 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:30.031074 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" Oct 31 00:50:30.031328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:30.031114 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55" event={"ID":"5bfb90df-62a9-4acd-afc1-3df560652008","Type":"ContainerDied","Data":"20d7fa71533762fa625c3116c0efbe2f15d7373d068b65cfd9a64f5823c182f1"} Oct 31 00:50:30.031328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:30.031223 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d7fa71533762fa625c3116c0efbe2f15d7373d068b65cfd9a64f5823c182f1" Oct 31 00:50:30.485344 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:30.485253 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:31.975401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:31.975364 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.181908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:32.181861 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:34.000678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.000649 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn"] Oct 31 00:50:34.001172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.001159 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5bfb90df-62a9-4acd-afc1-3df560652008" containerName="revoke-keys" Oct 31 00:50:34.001268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.001207 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfb90df-62a9-4acd-afc1-3df560652008" containerName="revoke-keys" Oct 31 00:50:34.001307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.001296 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfb90df-62a9-4acd-afc1-3df560652008" containerName="revoke-keys" Oct 31 00:50:34.005396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.005371 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.057591 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.057557 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-869cca-openshift-default-dockercfg-49wg8\"" Oct 31 00:50:34.101674 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.101639 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.101869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.101696 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.101869 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.101818 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.101952 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.101901 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.101989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.101967 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.102023 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.102002 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.102069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.102053 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.102204 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.102159 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.102260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.102244 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxh88\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.111922 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.111891 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn"] Oct 31 00:50:34.203967 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.203922 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204040 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204162 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204136 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204164 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxh88\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204244 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204285 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204344 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204392 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204490 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204438 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.204646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204487 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.205287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.204993 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.205287 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.205060 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.205501 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.205379 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.206255 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.206231 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.208863 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.207848 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.209527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.209501 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.265496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.265468 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.265688 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.265654 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxh88\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.321269 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.321235 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:34.763815 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.763786 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn"] Oct 31 00:50:34.765195 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:50:34.765153 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060262bf_3ba9_4455_ab8a_d05061a833bf.slice/crio-d689ae0d74139144f858fc867613b0ab6a72f1166f02c1de60476e01cfae4fbe WatchSource:0}: Error finding container d689ae0d74139144f858fc867613b0ab6a72f1166f02c1de60476e01cfae4fbe: Status 404 returned error can't find the container with id d689ae0d74139144f858fc867613b0ab6a72f1166f02c1de60476e01cfae4fbe Oct 31 00:50:34.767580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.767546 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:50:34.767638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.767614 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:50:34.767677 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:34.767643 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:50:35.065153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.065054 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" event={"ID":"060262bf-3ba9-4455-ab8a-d05061a833bf","Type":"ContainerStarted","Data":"a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96"} Oct 31 00:50:35.065153 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.065123 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" event={"ID":"060262bf-3ba9-4455-ab8a-d05061a833bf","Type":"ContainerStarted","Data":"d689ae0d74139144f858fc867613b0ab6a72f1166f02c1de60476e01cfae4fbe"} Oct 31 00:50:35.185420 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.185363 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podStartSLOduration=2.185348175 podStartE2EDuration="2.185348175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:35.184010777 +0000 UTC m=+2815.277301026" watchObservedRunningTime="2026-10-31 00:50:35.185348175 +0000 UTC m=+2815.278638362" Oct 31 00:50:35.321929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.321833 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:35.323434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.323408 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:35.323581 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.323480 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:35.554266 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.554152 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:35.703594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.703505 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:35.864215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:35.863963 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:36.321573 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:36.321542 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:36.322050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:36.321594 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:37.321938 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:37.321906 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:37.322477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:37.321965 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:37.485476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:37.485442 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:38.164809 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:38.164772 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:38.321523 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:38.321493 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:38.321698 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:38.321543 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:38.602610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:38.602574 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:39.052205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:39.049572 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf"] Oct 31 00:50:39.095653 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:39.095615 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af9cdb4-wrmvf"] Oct 31 00:50:39.322057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:39.321974 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:39.322057 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:39.322037 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:40.060558 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:40.060525 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63f1ca9-4fba-4544-a192-fabb3fbaedcf" path="/var/lib/kubelet/pods/e63f1ca9-4fba-4544-a192-fabb3fbaedcf/volumes" Oct 31 00:50:40.321590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:40.321504 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:40.321590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:40.321556 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:40.782966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:40.782923 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:41.321567 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.321533 5943 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:50:41.321993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.321587 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:50:41.413487 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.413402 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7"] Oct 31 00:50:41.418485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.418464 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.470873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.470836 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0144bd-openshift-default-dockercfg-4hs4v\"" Oct 31 00:50:41.518082 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518049 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518082 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518085 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntn4f\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518112 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518297 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518276 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518310 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518387 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518342 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518424 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518519 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518486 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.518610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.518527 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.522933 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.522908 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7"] Oct 31 00:50:41.619759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.619717 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.619972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.619773 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntn4f\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.619972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.619800 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.619972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.619874 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.619972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.619914 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.619972 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.619966 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620015 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620050 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620260 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620081 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620453 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620496 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620813 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620785 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.620951 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.620925 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.621080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.621055 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.623687 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.623661 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.624948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.624930 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.681004 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.680917 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.681290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.681256 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntn4f\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:41.731831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:41.731798 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:42.173667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.173637 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7"] Oct 31 00:50:42.174634 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:50:42.174603 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7345c8aa_6635_4a7f_9b14_079995d6c771.slice/crio-0fb2a925f294a04db035948e54233e89915af54df537f12e9d5ab3efefb9c80e WatchSource:0}: Error finding container 0fb2a925f294a04db035948e54233e89915af54df537f12e9d5ab3efefb9c80e: Status 404 returned error can't find the container with id 0fb2a925f294a04db035948e54233e89915af54df537f12e9d5ab3efefb9c80e Oct 31 00:50:42.177161 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.177126 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:50:42.177275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.177245 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:50:42.177347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.177290 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:50:42.326941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.326911 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:42.327478 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.327385 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:42.328276 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.328249 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:50:42.457606 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.457496 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.681759 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:42.681722 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.158436 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:43.158393 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" event={"ID":"7345c8aa-6635-4a7f-9b14-079995d6c771","Type":"ContainerStarted","Data":"93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e"} Oct 31 00:50:43.158645 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:43.158467 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" event={"ID":"7345c8aa-6635-4a7f-9b14-079995d6c771","Type":"ContainerStarted","Data":"0fb2a925f294a04db035948e54233e89915af54df537f12e9d5ab3efefb9c80e"} Oct 31 00:50:43.281893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:43.281827 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podStartSLOduration=2.281812346 podStartE2EDuration="2.281812346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:43.279603044 +0000 UTC m=+2823.372893256" watchObservedRunningTime="2026-10-31 00:50:43.281812346 +0000 UTC m=+2823.375102532" Oct 31 00:50:43.748202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:43.736520 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:43.753206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:43.749370 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:50:43.753206 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:43.749435 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 00:50:44.749208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:44.743447 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:50:44.749208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:44.743516 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 00:50:45.654485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:45.654445 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:45.735528 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:45.735495 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:50:45.735819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:45.735789 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 00:50:45.854003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:45.853959 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.021688 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:46.021653 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.242207 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:46.238869 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.388300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:46.388213 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.743265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:46.734767 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:50:46.743265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:46.734866 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 00:50:47.732632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:47.732596 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:50:47.733294 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:47.733260 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 00:50:48.733007 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:48.732970 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:50:48.733400 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:48.733053 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 31 00:50:50.735982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:50.735922 5943 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:50:50.736485 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:50.736005 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:50.797551 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:50.797511 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:50.799697 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:50.799674 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:50.800831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:50.800809 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:50:51.457046 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:51.457005 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.358489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:54.358453 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:56.050703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:56.050633 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4"] Oct 31 00:50:56.099827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:56.099795 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449ae58d-cqvm4"] Oct 31 00:50:56.452672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:56.452581 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:57.476904 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:50:57.476859 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:50:58.041948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:50:58.041915 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021a6b2a-5157-44ba-9a15-43d47223b3ff" path="/var/lib/kubelet/pods/021a6b2a-5157-44ba-9a15-43d47223b3ff/volumes" Oct 31 00:51:01.247389 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:01.247347 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:06.047671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:06.047636 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.850093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:10.850051 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.878887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:10.878859 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:51:11.792989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:11.792955 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:12.049300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:12.049219 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs"] Oct 31 00:51:12.098503 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:12.098470 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e06782c1-rjzcs"] Oct 31 00:51:14.044979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:14.044934 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d" path="/var/lib/kubelet/pods/f9c8d186-71d3-46cb-8b98-a3dcc6a04d8d/volumes" Oct 31 00:51:14.882563 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:14.882518 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.383760 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.383730 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.784468 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.784420 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.937136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.937105 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449"] Oct 31 00:51:15.940983 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.940961 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.945761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.945738 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.975111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975064 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975144 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975247 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6q2m\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975301 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975355 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975415 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975461 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975624 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975569 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:15.975624 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:15.975612 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.012767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.012734 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449"] Oct 31 00:51:16.043043 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.042954 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c6b3b9-openshift-default-dockercfg-pfsx7\"" Oct 31 00:51:16.077706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077673 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.077896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077714 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.077896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077745 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.077896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077783 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.077896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077858 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6q2m\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.077896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077894 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077935 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.077983 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.078027 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.078242 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078392 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.078320 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.078706 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078741 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.078733 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.078971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.078783 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.080566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.080542 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.080721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.080704 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.121444 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.121410 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.145763 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.144483 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.147172 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.147093 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6q2m\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m\") pod \"e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.257879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.257838 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:16.279339 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.279304 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.434881 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.434841 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.695040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.695009 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449"] Oct 31 00:51:16.695986 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:51:16.695962 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4584d7_e974_443c_955c_14e36f5a14fb.slice/crio-5f00ddff1ee13481bb88b71d36788113549d69077c5d970796f9b1de9b947f87 WatchSource:0}: Error finding container 5f00ddff1ee13481bb88b71d36788113549d69077c5d970796f9b1de9b947f87: Status 404 returned error can't find the container with id 5f00ddff1ee13481bb88b71d36788113549d69077c5d970796f9b1de9b947f87 Oct 31 00:51:16.698434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.698400 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:51:16.698541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.698478 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:51:16.698541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:16.698522 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:51:17.397907 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:17.397870 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" event={"ID":"0a4584d7-e974-443c-955c-14e36f5a14fb","Type":"ContainerStarted","Data":"9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71"} Oct 31 00:51:17.398090 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:17.397928 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" event={"ID":"0a4584d7-e974-443c-955c-14e36f5a14fb","Type":"ContainerStarted","Data":"5f00ddff1ee13481bb88b71d36788113549d69077c5d970796f9b1de9b947f87"} Oct 31 00:51:17.543055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:17.543000 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podStartSLOduration=2.5429856969999998 podStartE2EDuration="2.542985697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:17.540365375 +0000 UTC m=+2857.633655557" watchObservedRunningTime="2026-10-31 00:51:17.542985697 +0000 UTC m=+2857.636275884" Oct 31 00:51:18.258480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:18.258444 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:18.264111 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:18.264084 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:18.414413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:18.414374 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:18.415300 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:18.415279 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:51:18.683788 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:18.683700 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.163470 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:19.163430 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.785784 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:19.785747 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.094772 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:20.094684 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.467322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:20.467233 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.638360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:20.638328 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.789608 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:20.789577 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.459035 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:21.458995 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.046995 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.046910 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.227450 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.227423 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.454845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.454739 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.636243 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.636204 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.638100 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.638073 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn"] Oct 31 00:51:26.648491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.648462 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.694734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.694698 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn"] Oct 31 00:51:26.729982 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.729895 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.730157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.730015 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rn7q\" (UniqueName: \"kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q\") pod \"payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.739919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.739896 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c6b3b9\"" Oct 31 00:51:26.739919 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.739896 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c6b3b9-dockercfg-fcrjn\"" Oct 31 00:51:26.798553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.798526 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.830858 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.830829 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.831014 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.830897 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn7q\" (UniqueName: \"kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q\") pod \"payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.831526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.831505 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.891795 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.891764 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn7q\" (UniqueName: \"kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q\") pod \"payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:26.964576 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:26.964548 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:27.344151 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:27.344122 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn"] Oct 31 00:51:27.345888 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:51:27.345831 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1e350c_f3a8_4cee_91ad_7864f9ad7648.slice/crio-17a9b1288e646d79e0a2249be6d632b6535f27e55b982be59a76c8ffdb068a1e WatchSource:0}: Error finding container 17a9b1288e646d79e0a2249be6d632b6535f27e55b982be59a76c8ffdb068a1e: Status 404 returned error can't find the container with id 17a9b1288e646d79e0a2249be6d632b6535f27e55b982be59a76c8ffdb068a1e Oct 31 00:51:27.476157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:27.476121 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" event={"ID":"5c1e350c-f3a8-4cee-91ad-7864f9ad7648","Type":"ContainerStarted","Data":"b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278"} Oct 31 00:51:27.476377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:27.476203 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:27.476377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:27.476227 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" event={"ID":"5c1e350c-f3a8-4cee-91ad-7864f9ad7648","Type":"ContainerStarted","Data":"17a9b1288e646d79e0a2249be6d632b6535f27e55b982be59a76c8ffdb068a1e"} Oct 31 00:51:27.484397 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:51:27.484363 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:51:27.591030 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:27.590971 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" podStartSLOduration=1.5909498530000001 podStartE2EDuration="1.590949853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:27.590485607 +0000 UTC m=+2867.683775838" watchObservedRunningTime="2026-10-31 00:51:27.590949853 +0000 UTC m=+2867.684240041" Oct 31 00:51:29.048363 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:29.048324 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk"] Oct 31 00:51:29.098847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:29.098644 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63c123-2gblk"] Oct 31 00:51:30.040676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:30.040643 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c8f2be-e368-4bc7-99f7-574d0756afce" path="/var/lib/kubelet/pods/62c8f2be-e368-4bc7-99f7-574d0756afce/volumes" Oct 31 00:51:30.934640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:30.934602 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.128238 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:31.128164 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.519502 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:31.519464 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.837159 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:31.837070 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.985322 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:31.985291 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:36.564059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:36.564021 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.487945 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:38.487909 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 00:51:40.050397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:40.050363 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk"] Oct 31 00:51:40.096865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:40.096828 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2258a4-zwfsk"] Oct 31 00:51:41.137370 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:41.137337 5943 scope.go:122] "RemoveContainer" containerID="e5b126671ea85d5a3490483edb5e422f1bdcd21218c2c45e625c6b67e2b3a587" Oct 31 00:51:41.351060 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:41.351018 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:42.041882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:42.041849 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0fd701-cede-44f9-b620-d34c8a4de30a" path="/var/lib/kubelet/pods/2e0fd701-cede-44f9-b620-d34c8a4de30a/volumes" Oct 31 00:51:46.159659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:46.159620 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:47.416616 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:47.416583 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:47.875236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:47.875201 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:48.585618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:48.585580 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:48.875689 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:48.875556 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:51.266313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:51.266272 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:52.711063 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:52.711025 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:52.918434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:52.918397 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:54.698307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:54.698278 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:54.845332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:54.845300 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:51:55.928794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:55.928756 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:56.058222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.058007 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86"] Oct 31 00:51:56.089577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.089545 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:56.103354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.103325 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2c2a56-djx86"] Oct 31 00:51:56.325335 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.325303 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc"] Oct 31 00:51:56.331092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.331075 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.384755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.384717 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-bf325c\"" Oct 31 00:51:56.385019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.384881 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-bf325c-dockercfg-xz6gp\"" Oct 31 00:51:56.388226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.388164 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc"] Oct 31 00:51:56.440237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.432736 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.440237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.432822 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjnf\" (UniqueName: \"kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf\") pod \"payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.534402 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.534360 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.534602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.534423 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjnf\" (UniqueName: \"kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf\") pod \"payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.535341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.535312 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.551641 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.551590 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:56.596680 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.596605 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjnf\" (UniqueName: \"kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf\") pod \"payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:56.645598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:56.645560 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:57.022232 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:57.022202 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc"] Oct 31 00:51:57.024047 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:51:57.024019 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61de143f_81b8_4cd1_b5b6_5f5dd780c071.slice/crio-d1d396d1f928c3cc02f3bf39066e00fd385893c10ec5f3c61cdd27d1e001641e WatchSource:0}: Error finding container d1d396d1f928c3cc02f3bf39066e00fd385893c10ec5f3c61cdd27d1e001641e: Status 404 returned error can't find the container with id d1d396d1f928c3cc02f3bf39066e00fd385893c10ec5f3c61cdd27d1e001641e Oct 31 00:51:57.477506 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:51:57.477407 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:51:57.696755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:57.696715 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" event={"ID":"61de143f-81b8-4cd1-b5b6-5f5dd780c071","Type":"ContainerStarted","Data":"bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688"} Oct 31 00:51:57.697024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:57.696774 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" event={"ID":"61de143f-81b8-4cd1-b5b6-5f5dd780c071","Type":"ContainerStarted","Data":"d1d396d1f928c3cc02f3bf39066e00fd385893c10ec5f3c61cdd27d1e001641e"} Oct 31 00:51:57.697024 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:57.696817 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:51:57.813692 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:57.813634 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" podStartSLOduration=1.813619116 podStartE2EDuration="1.813619116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:57.812041051 +0000 UTC m=+2897.905331257" watchObservedRunningTime="2026-10-31 00:51:57.813619116 +0000 UTC m=+2897.906909327" Oct 31 00:51:58.040917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:51:58.040879 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28ac4da-7e56-4c4c-b59a-7f2d7fce1786" path="/var/lib/kubelet/pods/d28ac4da-7e56-4c4c-b59a-7f2d7fce1786/volumes" Oct 31 00:52:01.604324 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:01.604280 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:06.634803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:06.634769 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:08.707157 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:08.707126 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 00:52:11.048215 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:11.048169 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj"] Oct 31 00:52:11.097764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:11.097732 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515a7592-m6fqj"] Oct 31 00:52:11.600408 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:11.600370 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:12.049108 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:12.049059 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c3eb58-c529-4782-934b-47ef69770a57" path="/var/lib/kubelet/pods/b8c3eb58-c529-4782-934b-47ef69770a57/volumes" Oct 31 00:52:16.383727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:16.383689 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:16.576437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:16.576407 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:16.719407 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:16.719302 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:18.053459 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:18.053421 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p"] Oct 31 00:52:18.102526 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:18.102487 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f3f39-88l9p"] Oct 31 00:52:18.758025 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:18.757991 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:19.230793 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.230702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:19.765649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.765572 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h"] Oct 31 00:52:19.770777 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.770752 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.856143 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.856090 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h"] Oct 31 00:52:19.864580 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.864404 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:19.893115 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.893052 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-093220-openshift-default-dockercfg-b8knl\"" Oct 31 00:52:19.924091 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924058 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924121 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924245 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924276 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924313 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2f8\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924334 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924443 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924543 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:19.924609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:19.924588 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025472 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025584 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025640 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025722 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025703 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025746 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025789 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025820 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025874 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2f8\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.025910 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.025904 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.026198 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.026148 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.026268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.026236 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.026327 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.026286 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.026424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.026405 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.026566 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.026542 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.028196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.028154 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.028854 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.028836 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.041994 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.041966 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d9e02f-53a2-4725-a451-a457a0984683" path="/var/lib/kubelet/pods/76d9e02f-53a2-4725-a451-a457a0984683/volumes" Oct 31 00:52:20.086579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.086545 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.086934 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.086911 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2f8\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8\") pod \"e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.087862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.087846 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:20.302587 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.302486 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.524310 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.524283 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h"] Oct 31 00:52:20.525910 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:52:20.525875 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b91f93_ae8e_418c_ac6f_53a444084621.slice/crio-2ea6a482f21a7999efde8b45d228c66fb8e843ec91bea2b728d54473a397d2f4 WatchSource:0}: Error finding container 2ea6a482f21a7999efde8b45d228c66fb8e843ec91bea2b728d54473a397d2f4: Status 404 returned error can't find the container with id 2ea6a482f21a7999efde8b45d228c66fb8e843ec91bea2b728d54473a397d2f4 Oct 31 00:52:20.528721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.528684 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:52:20.528798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.528745 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:52:20.528798 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.528778 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:52:20.878424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.878313 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" event={"ID":"62b91f93-ae8e-418c-ac6f-53a444084621","Type":"ContainerStarted","Data":"4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb"} Oct 31 00:52:20.878424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:20.878387 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" event={"ID":"62b91f93-ae8e-418c-ac6f-53a444084621","Type":"ContainerStarted","Data":"2ea6a482f21a7999efde8b45d228c66fb8e843ec91bea2b728d54473a397d2f4"} Oct 31 00:52:21.000801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:21.000746 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podStartSLOduration=2.000732383 podStartE2EDuration="2.000732383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:20.997671689 +0000 UTC m=+2921.090961879" watchObservedRunningTime="2026-10-31 00:52:21.000732383 +0000 UTC m=+2921.094022570" Oct 31 00:52:21.088736 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:21.088701 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:21.094307 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:21.094284 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:21.669274 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:21.669234 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:21.886228 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:21.886189 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:21.887224 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:21.887170 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:52:23.890337 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:23.890305 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:24.093586 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:24.093550 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:24.268659 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:24.268623 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:26.713667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:26.713629 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:26.918649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:26.918609 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:27.067672 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:27.067637 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:27.477293 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:52:27.477165 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:52:31.269981 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.269943 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:31.465437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.465403 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:31.843332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.843302 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf"] Oct 31 00:52:31.848865 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.848841 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:31.873905 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.873874 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:31.874067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.873991 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsw6\" (UniqueName: \"kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6\") pod \"payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:31.901684 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.901652 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-093220\"" Oct 31 00:52:31.951141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.951101 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-093220-dockercfg-h8jk6\"" Oct 31 00:52:31.954134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.954110 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf"] Oct 31 00:52:31.975137 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.975103 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:31.975317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.975161 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsw6\" (UniqueName: \"kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6\") pod \"payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:31.975814 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.975792 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:31.978102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:31.978070 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:32.037598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:32.037535 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsw6\" (UniqueName: \"kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6\") pod \"payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:32.166222 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:32.166112 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:32.544794 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:32.544764 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf"] Oct 31 00:52:32.546633 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:52:32.546598 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419e6b2d_8266_46d8_8589_7daa5d359bf6.slice/crio-14b3529e54375f1009342c25d83c253c731f1d55fde617c7221c3a8676e05f3e WatchSource:0}: Error finding container 14b3529e54375f1009342c25d83c253c731f1d55fde617c7221c3a8676e05f3e: Status 404 returned error can't find the container with id 14b3529e54375f1009342c25d83c253c731f1d55fde617c7221c3a8676e05f3e Oct 31 00:52:32.964810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:32.964721 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" event={"ID":"419e6b2d-8266-46d8-8589-7daa5d359bf6","Type":"ContainerStarted","Data":"09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58"} Oct 31 00:52:32.964810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:32.964775 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" event={"ID":"419e6b2d-8266-46d8-8589-7daa5d359bf6","Type":"ContainerStarted","Data":"14b3529e54375f1009342c25d83c253c731f1d55fde617c7221c3a8676e05f3e"} Oct 31 00:52:32.964810 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:32.964803 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:33.049974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:33.049942 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc"] Oct 31 00:52:33.069147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:33.069116 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fca6de8-5m9mc"] Oct 31 00:52:33.097965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:33.097913 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" podStartSLOduration=2.097899195 podStartE2EDuration="2.097899195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:33.080266531 +0000 UTC m=+2933.173556745" watchObservedRunningTime="2026-10-31 00:52:33.097899195 +0000 UTC m=+2933.191189406" Oct 31 00:52:34.042212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:34.042149 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce8b42f-fccc-456a-bdd7-b582e081ddd5" path="/var/lib/kubelet/pods/fce8b42f-fccc-456a-bdd7-b582e081ddd5/volumes" Oct 31 00:52:36.749979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:36.749942 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:41.164683 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.164652 5943 scope.go:122] "RemoveContainer" containerID="14efe71221527ddf24dae571334376747a681249abae709b9f3283f69481fd44" Oct 31 00:52:41.174937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.174914 5943 scope.go:122] "RemoveContainer" containerID="0fe5e4d600feae3d229556e4ec8ab7b348d2cfc0501832d732ddf6d5a2001d13" Oct 31 00:52:41.184094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.184075 5943 scope.go:122] "RemoveContainer" containerID="b39800d40b8229ed2c3a94275ff4cc08de71e14ba0d963ff428290851b8092a2" Oct 31 00:52:41.193928 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.193902 5943 scope.go:122] "RemoveContainer" containerID="0867a6fb230d61134aa23669c1930b0dc4757008e5066d8c54b4932ba3212cbb" Oct 31 00:52:41.205268 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.205245 5943 scope.go:122] "RemoveContainer" containerID="f678c0aa8a04f98aa8a489dad8a97302605997759fa7f357379779303c4c8d2b" Oct 31 00:52:41.544851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.544810 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:41.741448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:41.741417 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:52:43.975477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:43.975444 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 00:52:45.259852 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:45.259783 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:46.798538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:46.798501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.603332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.603300 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc"] Oct 31 00:52:48.610019 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.609995 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.611920 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.611886 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.612064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.611957 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw827\" (UniqueName: \"kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.612145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.612110 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.612226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.612213 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.612346 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.612328 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.612411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.612381 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.665734 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.665695 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc"] Oct 31 00:52:48.697959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.697918 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.711443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.711415 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1a1181\"/\"kube-root-ca.crt\"" Oct 31 00:52:48.711629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.711484 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1a1181\"/\"default-dockercfg-z4c97\"" Oct 31 00:52:48.711629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.711487 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1a1181\"/\"openshift-service-ca.crt\"" Oct 31 00:52:48.711629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.711598 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1a1181\"/\"autores-model-1a1181-kserve-self-signed-certs\"" Oct 31 00:52:48.713351 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.713328 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.713460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.713400 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.713529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.713462 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.713529 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.713510 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw827\" (UniqueName: \"kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.713637 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.713595 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.713700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.713661 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.714069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.714037 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.714375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.714086 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.714375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.714263 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.716627 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.716603 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.717002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.716976 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.776118 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.776073 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw827\" (UniqueName: \"kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827\") pod \"autores-model-1a1181-kserve-db6f5dbf7-jrhqc\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:48.925979 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:48.925888 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:49.332342 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:49.332312 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc"] Oct 31 00:52:49.332740 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:52:49.332718 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9daa709d_ddad_4f04_b396_be419019de6b.slice/crio-93d3efd5704de6c55c2a0b8b8ca07ded9f0855ae111c61aaac462480f4ec1ff0 WatchSource:0}: Error finding container 93d3efd5704de6c55c2a0b8b8ca07ded9f0855ae111c61aaac462480f4ec1ff0: Status 404 returned error can't find the container with id 93d3efd5704de6c55c2a0b8b8ca07ded9f0855ae111c61aaac462480f4ec1ff0 Oct 31 00:52:50.130239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:50.130149 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" event={"ID":"9daa709d-ddad-4f04-b396-be419019de6b","Type":"ContainerStarted","Data":"340e3b8f2fc549da60a34bebe000286b4da5fe518e6a77e54773f1554c427301"} Oct 31 00:52:50.130719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:50.130279 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" event={"ID":"9daa709d-ddad-4f04-b396-be419019de6b","Type":"ContainerStarted","Data":"93d3efd5704de6c55c2a0b8b8ca07ded9f0855ae111c61aaac462480f4ec1ff0"} Oct 31 00:52:50.249649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:50.249593 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" podStartSLOduration=2.249579885 podStartE2EDuration="2.249579885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:50.247153606 +0000 UTC m=+2950.340443816" watchObservedRunningTime="2026-10-31 00:52:50.249579885 +0000 UTC m=+2950.342870072" Oct 31 00:52:50.926540 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:50.926511 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:51.940522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:51.940483 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:52.145233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:52.145123 5943 generic.go:356] "Generic (PLEG): container finished" podID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerID="9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71" exitCode=0 Oct 31 00:52:52.145233 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:52.145212 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" event={"ID":"0a4584d7-e974-443c-955c-14e36f5a14fb","Type":"ContainerDied","Data":"9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71"} Oct 31 00:52:52.145706 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:52.145689 5943 scope.go:122] "RemoveContainer" containerID="9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71" Oct 31 00:52:52.147245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:52.147218 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:52:52.147359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:52.147301 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:52:52.147359 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:52.147351 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:52:53.159686 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:53.159651 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" event={"ID":"0a4584d7-e974-443c-955c-14e36f5a14fb","Type":"ContainerStarted","Data":"aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6"} Oct 31 00:52:53.258500 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:53.258461 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:52:53.258800 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:53.258775 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:52:53.258891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:53.258830 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:52:54.258521 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:54.258483 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:52:54.258964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:54.258550 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:52:55.053911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:55.053828 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:55.259052 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:55.259012 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:52:55.259477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:55.259093 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:52:56.062604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:56.062566 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv"] Oct 31 00:52:56.103049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:56.103013 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d59b5af2-qhknv"] Oct 31 00:52:56.259574 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:56.259539 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:52:56.260097 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:56.259588 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:52:56.760447 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:56.760408 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:57.258697 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:57.258662 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:52:57.258900 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:57.258732 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:52:57.481062 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:52:57.481021 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:52:58.057913 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:58.057880 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9646978-d03a-4013-ab69-9fbb4a5aeda7" path="/var/lib/kubelet/pods/e9646978-d03a-4013-ab69-9fbb4a5aeda7/volumes" Oct 31 00:52:58.259262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:58.259228 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:52:58.259443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:58.259317 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:52:58.926086 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:58.926051 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:58.939003 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:58.938973 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:59.210491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:59.210407 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:52:59.263550 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:59.263521 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:52:59.263831 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:59.263807 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:52:59.264437 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:52:59.264421 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 00:53:01.424924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:01.424878 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:01.624527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:01.624478 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:01.775332 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:01.775295 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:06.381497 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:06.381451 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:10.759477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:10.759434 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:53:11.283818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:11.283783 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:16.050112 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:16.050078 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t"] Oct 31 00:53:16.052598 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:16.052556 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:16.099887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:16.099851 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb6f9a5-pqv8t"] Oct 31 00:53:18.041585 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:18.041544 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b97fa9-f534-4947-896a-147fb4f91e4f" path="/var/lib/kubelet/pods/36b97fa9-f534-4947-896a-147fb4f91e4f/volumes" Oct 31 00:53:20.784962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:20.784872 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:25.508761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:25.508721 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:27.475683 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:53:27.475643 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:53:30.243170 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:30.243127 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:31.049150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:31.049113 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj"] Oct 31 00:53:31.098141 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:31.098106 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cc1d26-wnnnj"] Oct 31 00:53:32.041801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:32.041764 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9" path="/var/lib/kubelet/pods/dd1bf2f6-ba69-4f1f-8e8a-2ecf350b28b9/volumes" Oct 31 00:53:33.645638 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:33.645602 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:35.209134 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:35.209087 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:35.365588 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:35.365558 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:35.513168 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:35.513134 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:40.275262 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:40.275221 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:41.261136 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:41.261106 5943 scope.go:122] "RemoveContainer" containerID="bf1fae566aab469c35a300f146a9d53284d586a4ec4eea7c611e5d3dc617a75c" Oct 31 00:53:41.271439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:41.271419 5943 scope.go:122] "RemoveContainer" containerID="efa13f51fa60252135fba1edb7cedc2dafa7a2ec4d62be94f50ad0b88ecd8d08" Oct 31 00:53:41.280775 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:41.280751 5943 scope.go:122] "RemoveContainer" containerID="7def9eb15eb00457bb2f4e96d9c43170d29d67dadd61f4f948c0252a16568a3a" Oct 31 00:53:41.290411 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:41.290391 5943 scope.go:122] "RemoveContainer" containerID="cccf8c611398ba2f264a52db4af1a121f2be457ae58142770170d41b0cd6a458" Oct 31 00:53:41.299078 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:41.299059 5943 scope.go:122] "RemoveContainer" containerID="bd66e620123c94033a5ce53c1982d8ab4e3739c003958c2e1fef91779d797953" Oct 31 00:53:44.566245 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:44.566205 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:45.071691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:45.071651 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:47.047552 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:47.047472 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf"] Oct 31 00:53:47.096619 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:47.096589 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3f536d-crlwf"] Oct 31 00:53:47.810079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:47.810046 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:48.041764 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:48.041731 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f37fe2f-7e84-4cf4-9878-51da1dfc8973" path="/var/lib/kubelet/pods/9f37fe2f-7e84-4cf4-9878-51da1dfc8973/volumes" Oct 31 00:53:48.540375 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:48.540347 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc"] Oct 31 00:53:48.589965 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:48.589921 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:49.064442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.064404 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p"] Oct 31 00:53:49.070462 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.070438 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.128860 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.128828 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p"] Oct 31 00:53:49.160584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.160558 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:53:49.160584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.160579 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:49.160584 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.160560 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:49.161339 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.161319 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:49.166102 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.166075 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.166275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.166249 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.166424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.166405 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlphp\" (UniqueName: \"kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.268365 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.268323 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlphp\" (UniqueName: \"kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.268565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.268475 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.268565 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.268533 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.269319 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.269297 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.270941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.270924 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.337787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.337719 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlphp\" (UniqueName: \"kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp\") pod \"maas-api-revoke-keys-e2e-autores-1a1181-5p28p\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.385252 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.385221 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:49.772325 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.772075 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p"] Oct 31 00:53:49.774691 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:53:49.774664 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f382899_e4ff_4136_9a9e_040f0dc7df4a.slice/crio-a6dfc2d5e2dd9ad1845f4add2a9e57152076093bd7229606719e04de5ac05139 WatchSource:0}: Error finding container a6dfc2d5e2dd9ad1845f4add2a9e57152076093bd7229606719e04de5ac05139: Status 404 returned error can't find the container with id a6dfc2d5e2dd9ad1845f4add2a9e57152076093bd7229606719e04de5ac05139 Oct 31 00:53:49.979212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:49.979167 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:53:50.182477 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:50.182434 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:50.554277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:50.554245 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" event={"ID":"0f382899-e4ff-4136-9a9e-040f0dc7df4a","Type":"ContainerStarted","Data":"4711018529bc9c922140481f3b0381d040089280ade1e8ffb19fe13a78820c06"} Oct 31 00:53:50.554484 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:50.554331 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" event={"ID":"0f382899-e4ff-4136-9a9e-040f0dc7df4a","Type":"ContainerStarted","Data":"a6dfc2d5e2dd9ad1845f4add2a9e57152076093bd7229606719e04de5ac05139"} Oct 31 00:53:50.671835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:50.671786 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" podStartSLOduration=1.671773261 podStartE2EDuration="1.671773261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:50.669667596 +0000 UTC m=+3010.762957805" watchObservedRunningTime="2026-10-31 00:53:50.671773261 +0000 UTC m=+3010.765063485" Oct 31 00:53:51.037379 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:51.037338 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:51.563624 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:51.563586 5943 generic.go:356] "Generic (PLEG): container finished" podID="0f382899-e4ff-4136-9a9e-040f0dc7df4a" containerID="4711018529bc9c922140481f3b0381d040089280ade1e8ffb19fe13a78820c06" exitCode=0 Oct 31 00:53:51.563816 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:51.563634 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" event={"ID":"0f382899-e4ff-4136-9a9e-040f0dc7df4a","Type":"ContainerDied","Data":"4711018529bc9c922140481f3b0381d040089280ade1e8ffb19fe13a78820c06"} Oct 31 00:53:52.758210 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.758145 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:52.820944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.820855 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp\" (UniqueName: \"kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp\") pod \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " Oct 31 00:53:52.820944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.820933 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca\") pod \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " Oct 31 00:53:52.821221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.820995 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca\") pod \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\" (UID: \"0f382899-e4ff-4136-9a9e-040f0dc7df4a\") " Oct 31 00:53:52.821407 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.821378 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca" pod "0f382899-e4ff-4136-9a9e-040f0dc7df4a" (UID: "0f382899-e4ff-4136-9a9e-040f0dc7df4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:52.821713 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.821656 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:53:52.823326 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.823293 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp" pod "0f382899-e4ff-4136-9a9e-040f0dc7df4a" (UID: "0f382899-e4ff-4136-9a9e-040f0dc7df4a"). InnerVolumeSpecName "kube-api-access-dlphp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:52.823435 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.823375 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca" pod "0f382899-e4ff-4136-9a9e-040f0dc7df4a" (UID: "0f382899-e4ff-4136-9a9e-040f0dc7df4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:52.923302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.923264 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f382899-e4ff-4136-9a9e-040f0dc7df4a-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:53:52.923302 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:52.923293 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlphp\" (UniqueName: \"kubernetes.io/projected/0f382899-e4ff-4136-9a9e-040f0dc7df4a-kube-api-access-dlphp\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:53:53.577685 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:53.577652 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" Oct 31 00:53:53.577882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:53.577684 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p" event={"ID":"0f382899-e4ff-4136-9a9e-040f0dc7df4a","Type":"ContainerDied","Data":"a6dfc2d5e2dd9ad1845f4add2a9e57152076093bd7229606719e04de5ac05139"} Oct 31 00:53:53.577882 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:53.577738 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6dfc2d5e2dd9ad1845f4add2a9e57152076093bd7229606719e04de5ac05139" Oct 31 00:53:55.282667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:55.282629 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:55.478941 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:53:55.478905 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:57.477407 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:53:57.477367 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:54:00.103670 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.103609 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:00.250943 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.250909 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:00.402906 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.402869 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:00.628976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.628881 5943 generic.go:356] "Generic (PLEG): container finished" podID="62b91f93-ae8e-418c-ac6f-53a444084621" containerID="4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb" exitCode=0 Oct 31 00:54:00.628976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.628956 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" event={"ID":"62b91f93-ae8e-418c-ac6f-53a444084621","Type":"ContainerDied","Data":"4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb"} Oct 31 00:54:00.629491 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.629472 5943 scope.go:122] "RemoveContainer" containerID="4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb" Oct 31 00:54:00.631317 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.631286 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:54:00.631442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.631359 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:54:00.631442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:00.631398 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 00:54:01.674059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:01.672383 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" event={"ID":"62b91f93-ae8e-418c-ac6f-53a444084621","Type":"ContainerStarted","Data":"127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68"} Oct 31 00:54:02.088442 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:02.088405 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:54:02.088691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:02.088669 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 00:54:02.088761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:02.088718 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 00:54:03.088732 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.088692 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 00:54:03.089199 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.088768 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 00:54:03.562847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.562796 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" podUID="9daa709d-ddad-4f04-b396-be419019de6b" containerName="main" containerID="cri-o://340e3b8f2fc549da60a34bebe000286b4da5fe518e6a77e54773f1554c427301" gracePeriod=45 Oct 31 00:54:03.687848 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.687818 5943 generic.go:356] "Generic (PLEG): container finished" podID="9daa709d-ddad-4f04-b396-be419019de6b" containerID="340e3b8f2fc549da60a34bebe000286b4da5fe518e6a77e54773f1554c427301" exitCode=0 Oct 31 00:54:03.688002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.687888 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" event={"ID":"9daa709d-ddad-4f04-b396-be419019de6b","Type":"ContainerDied","Data":"340e3b8f2fc549da60a34bebe000286b4da5fe518e6a77e54773f1554c427301"} Oct 31 00:54:03.868265 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.868237 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:54:03.892496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892464 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm\") pod \"9daa709d-ddad-4f04-b396-be419019de6b\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " Oct 31 00:54:03.892676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892520 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir\") pod \"9daa709d-ddad-4f04-b396-be419019de6b\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " Oct 31 00:54:03.892676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892576 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home\") pod \"9daa709d-ddad-4f04-b396-be419019de6b\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " Oct 31 00:54:03.892676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892631 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs\") pod \"9daa709d-ddad-4f04-b396-be419019de6b\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " Oct 31 00:54:03.892676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892650 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache\") pod \"9daa709d-ddad-4f04-b396-be419019de6b\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " Oct 31 00:54:03.892902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892692 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827\" (UniqueName: \"kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827\") pod \"9daa709d-ddad-4f04-b396-be419019de6b\" (UID: \"9daa709d-ddad-4f04-b396-be419019de6b\") " Oct 31 00:54:03.892902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892839 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir" pod "9daa709d-ddad-4f04-b396-be419019de6b" (UID: "9daa709d-ddad-4f04-b396-be419019de6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:03.892902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892893 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home" pod "9daa709d-ddad-4f04-b396-be419019de6b" (UID: "9daa709d-ddad-4f04-b396-be419019de6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:03.893064 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.892964 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache" pod "9daa709d-ddad-4f04-b396-be419019de6b" (UID: "9daa709d-ddad-4f04-b396-be419019de6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:03.893218 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.893171 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:03.893298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.893220 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:03.893298 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.893229 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:03.894896 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.894877 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm" pod "9daa709d-ddad-4f04-b396-be419019de6b" (UID: "9daa709d-ddad-4f04-b396-be419019de6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:03.895010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.894966 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827" pod "9daa709d-ddad-4f04-b396-be419019de6b" (UID: "9daa709d-ddad-4f04-b396-be419019de6b"). InnerVolumeSpecName "kube-api-access-pw827". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:03.895079 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.895034 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs" pod "9daa709d-ddad-4f04-b396-be419019de6b" (UID: "9daa709d-ddad-4f04-b396-be419019de6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:03.994856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.994821 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9daa709d-ddad-4f04-b396-be419019de6b-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:03.994856 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.994858 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9daa709d-ddad-4f04-b396-be419019de6b-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:03.995070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:03.994874 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw827\" (UniqueName: \"kubernetes.io/projected/9daa709d-ddad-4f04-b396-be419019de6b-kube-api-access-pw827\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:04.089239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.089110 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 00:54:04.089239 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.089169 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 00:54:04.695589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.695554 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" Oct 31 00:54:04.695817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.695556 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc" event={"ID":"9daa709d-ddad-4f04-b396-be419019de6b","Type":"ContainerDied","Data":"93d3efd5704de6c55c2a0b8b8ca07ded9f0855ae111c61aaac462480f4ec1ff0"} Oct 31 00:54:04.695817 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.695697 5943 scope.go:122] "RemoveContainer" containerID="340e3b8f2fc549da60a34bebe000286b4da5fe518e6a77e54773f1554c427301" Oct 31 00:54:04.854837 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.854798 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:04.865944 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.865914 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc"] Oct 31 00:54:04.869537 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:04.869508 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1a1181/autores-model-1a1181-kserve-db6f5dbf7-jrhqc"] Oct 31 00:54:05.088871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:05.088834 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 00:54:05.089037 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:05.088889 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 00:54:06.059767 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:06.059734 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9daa709d-ddad-4f04-b396-be419019de6b" path="/var/lib/kubelet/pods/9daa709d-ddad-4f04-b396-be419019de6b/volumes" Oct 31 00:54:06.089164 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:06.089131 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 00:54:06.089345 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:06.089211 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 00:54:06.886976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:06.886941 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:54:07.093067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:07.093037 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:54:07.718323 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:07.718290 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 00:54:09.728640 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:09.728605 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:11.046773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:11.046739 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b"] Oct 31 00:54:11.096128 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:11.096098 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3c7a1-n5g2b"] Oct 31 00:54:12.041649 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:12.041611 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e868180-588c-4084-85c5-9f6d4ac9c3ee" path="/var/lib/kubelet/pods/6e868180-588c-4084-85c5-9f6d4ac9c3ee/volumes" Oct 31 00:54:14.519762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:14.519724 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:19.375805 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:19.375755 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:20.159835 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:20.159791 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:54:21.312568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:21.312532 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:24.689589 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:24.689552 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:27.476436 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:54:27.476396 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:54:29.864220 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:29.864156 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:34.699973 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:34.699933 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:39.412894 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:39.412844 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:40.559873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:40.559835 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:41.344873 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.344843 5943 scope.go:122] "RemoveContainer" containerID="282fc84a2cfdec25219218f5cfa3cc30d2738286961917e3dbb319491b486115" Oct 31 00:54:41.354871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.354846 5943 scope.go:122] "RemoveContainer" containerID="37a090b1a8c1d9410d357d8e251b7968670730948d46935d6f5787a0295a2c1e" Oct 31 00:54:41.363655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.363631 5943 scope.go:122] "RemoveContainer" containerID="b38185a1e5145cfbfe37eee60fd27b04ba3a7e64cf93765e68e1e6aa8dfe67b1" Oct 31 00:54:41.373209 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.373163 5943 scope.go:122] "RemoveContainer" containerID="b416ca7d0eaf51cb361ceb0c7e252ff5cd54ccadf895ae435dabf0e5ff432bb9" Oct 31 00:54:41.381847 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.381819 5943 scope.go:122] "RemoveContainer" containerID="f9e1c93bf10bd429e45f6784d967453fe6750e17b403fd6a6aa28183b28dc8b2" Oct 31 00:54:41.393823 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.393804 5943 scope.go:122] "RemoveContainer" containerID="06a51186e09c3ecd2246c6d8df2299d787e828971d952890d1ed74010d9bdfd6" Oct 31 00:54:41.403110 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:41.403087 5943 scope.go:122] "RemoveContainer" containerID="84a27dc3844283ad8ea104ffba936bf3a3cd206c442fd8f9cfb5e87e159281c5" Oct 31 00:54:44.142208 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:44.142129 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:48.911002 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:48.910905 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:52.686012 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.685965 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx"] Oct 31 00:54:52.686778 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.686755 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9daa709d-ddad-4f04-b396-be419019de6b" containerName="main" Oct 31 00:54:52.686899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.686782 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daa709d-ddad-4f04-b396-be419019de6b" containerName="main" Oct 31 00:54:52.686899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.686794 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0f382899-e4ff-4136-9a9e-040f0dc7df4a" containerName="revoke-keys" Oct 31 00:54:52.686899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.686803 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f382899-e4ff-4136-9a9e-040f0dc7df4a" containerName="revoke-keys" Oct 31 00:54:52.687059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.686944 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daa709d-ddad-4f04-b396-be419019de6b" containerName="main" Oct 31 00:54:52.687059 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.686960 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f382899-e4ff-4136-9a9e-040f0dc7df4a" containerName="revoke-keys" Oct 31 00:54:52.692240 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.692215 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.724594 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.724553 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.724801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.724620 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.724801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.724747 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfzw\" (UniqueName: \"kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.724801 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.724771 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.724986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.724818 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.724986 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.724930 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.746665 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.746631 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f13553\"/\"openshift-service-ca.crt\"" Oct 31 00:54:52.782473 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.782430 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:52.795862 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.795830 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f13553\"/\"kube-root-ca.crt\"" Oct 31 00:54:52.796318 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.796292 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f13553\"/\"auth-test-model-f13553-kserve-self-signed-certs\"" Oct 31 00:54:52.796433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.796334 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f13553\"/\"default-dockercfg-86pmt\"" Oct 31 00:54:52.798821 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.798801 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx"] Oct 31 00:54:52.826433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826391 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.826647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826480 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.826647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826601 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfzw\" (UniqueName: \"kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.826647 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826639 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.826902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826710 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.826902 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826749 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.827065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.826935 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.827065 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.827064 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.827611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.827585 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.830410 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.830386 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.830538 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.830507 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:52.891015 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:52.890978 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfzw\" (UniqueName: \"kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw\") pod \"auth-test-model-f13553-kserve-645c49fb45-ckjpx\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:53.004509 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:53.004408 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:53.417908 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:53.417881 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx"] Oct 31 00:54:53.418513 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:54:53.418482 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod780306a8_3607_4c87_8343_f7824cca0b6a.slice/crio-c962dc4e6b0390384ef32d7283bb7e9d27ffcb4b251bb1a841be630d43f9923f WatchSource:0}: Error finding container c962dc4e6b0390384ef32d7283bb7e9d27ffcb4b251bb1a841be630d43f9923f: Status 404 returned error can't find the container with id c962dc4e6b0390384ef32d7283bb7e9d27ffcb4b251bb1a841be630d43f9923f Oct 31 00:54:53.686545 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:53.686443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:54.050757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:54.050717 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" event={"ID":"780306a8-3607-4c87-8343-f7824cca0b6a","Type":"ContainerStarted","Data":"e9d97bfe033e9f850057b58fae952782fb5af286b02c6f85a799d1bbc6271e1f"} Oct 31 00:54:54.050950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:54.050766 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" event={"ID":"780306a8-3607-4c87-8343-f7824cca0b6a","Type":"ContainerStarted","Data":"c962dc4e6b0390384ef32d7283bb7e9d27ffcb4b251bb1a841be630d43f9923f"} Oct 31 00:54:54.166924 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:54.166756 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" podStartSLOduration=2.166736234 podStartE2EDuration="2.166736234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:54.165813611 +0000 UTC m=+3074.259103827" watchObservedRunningTime="2026-10-31 00:54:54.166736234 +0000 UTC m=+3074.260026421" Oct 31 00:54:55.005277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:55.005242 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:54:55.751335 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:55.751299 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.971872 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:55.971833 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w"] Oct 31 00:54:55.975844 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:55.975824 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.054560 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.054529 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-16b984\"/\"openshift-service-ca.crt\"" Oct 31 00:54:56.071575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.071528 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.071575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.071582 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.071857 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.071740 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.071857 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.071809 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.071957 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.071865 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.072123 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.072102 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6scx4\" (UniqueName: \"kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.077460 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.077426 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-16b984\"/\"kube-root-ca.crt\"" Oct 31 00:54:56.077602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.077426 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-16b984\"/\"auth-test-model-16b984-kserve-self-signed-certs\"" Oct 31 00:54:56.077602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.077458 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-16b984\"/\"default-dockercfg-52ssj\"" Oct 31 00:54:56.077974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.077951 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:56.080423 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.080404 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w"] Oct 31 00:54:56.173145 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173105 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6scx4\" (UniqueName: \"kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173234 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173362 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173266 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173360 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173402 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173458 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173733 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173709 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173883 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173750 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.173948 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.173902 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.175899 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.175879 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.176077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.176060 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.235216 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.235150 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6scx4\" (UniqueName: \"kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4\") pod \"auth-test-model-16b984-kserve-5554cddd5-wbz2w\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.289866 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.289821 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:56.691845 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:56.691815 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w"] Oct 31 00:54:56.693708 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:54:56.693680 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3cda834_3d91_4ae7_a139_daaadce7d091.slice/crio-c1ae8c8b0def1c5c2878209696c3a9cfaa3150fed82d561eba6396e292cba67b WatchSource:0}: Error finding container c1ae8c8b0def1c5c2878209696c3a9cfaa3150fed82d561eba6396e292cba67b: Status 404 returned error can't find the container with id c1ae8c8b0def1c5c2878209696c3a9cfaa3150fed82d561eba6396e292cba67b Oct 31 00:54:57.073698 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:57.073648 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" event={"ID":"b3cda834-3d91-4ae7-a139-daaadce7d091","Type":"ContainerStarted","Data":"49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2"} Oct 31 00:54:57.073698 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:57.073701 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" event={"ID":"b3cda834-3d91-4ae7-a139-daaadce7d091","Type":"ContainerStarted","Data":"c1ae8c8b0def1c5c2878209696c3a9cfaa3150fed82d561eba6396e292cba67b"} Oct 31 00:54:57.191992 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:57.191938 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" podStartSLOduration=2.191921289 podStartE2EDuration="2.191921289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:57.189865966 +0000 UTC m=+3077.283156176" watchObservedRunningTime="2026-10-31 00:54:57.191921289 +0000 UTC m=+3077.285211520" Oct 31 00:54:57.289950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:57.289913 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:54:57.478172 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:54:57.478062 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:54:57.710989 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:57.710949 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:58.765406 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:54:58.765360 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:02.971949 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:02.971916 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:03.005221 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:03.005172 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:55:03.018792 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:03.018760 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:55:03.132568 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:03.132523 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 00:55:04.516727 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:04.516677 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:04.757303 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:04.757172 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:04.907578 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:04.907478 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:05.185827 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:05.185723 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p"] Oct 31 00:55:05.235849 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:05.235809 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1a1181-5p28p"] Oct 31 00:55:05.559243 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:05.559208 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:06.041629 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:06.041584 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f382899-e4ff-4136-9a9e-040f0dc7df4a" path="/var/lib/kubelet/pods/0f382899-e4ff-4136-9a9e-040f0dc7df4a/volumes" Oct 31 00:55:06.290008 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:06.289973 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:55:06.303010 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:06.302918 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:55:07.050439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:07.050405 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28"] Oct 31 00:55:07.052819 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:07.052794 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269cbaab-twv28"] Oct 31 00:55:07.159231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:07.159199 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 00:55:08.041976 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:08.041936 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8482d258-1d47-4a7b-beb0-bc274f3e302c" path="/var/lib/kubelet/pods/8482d258-1d47-4a7b-beb0-bc274f3e302c/volumes" Oct 31 00:55:08.900496 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:08.900453 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:55:09.864214 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:09.864163 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:11.893757 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:11.893722 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:12.050719 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:12.050680 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:12.287762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:12.287709 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:12.984092 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:12.984056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:13.726205 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:13.720646 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:14.075842 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:14.075802 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:14.765868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:14.765826 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.441808 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:15.441746 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.592236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:15.592202 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.746237 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:15.746137 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:19.654443 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:19.654406 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.445762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:24.445723 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.782771 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:24.782729 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.935248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:24.935212 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.475817 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:55:27.475773 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:55:29.245553 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:29.245515 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:30.047492 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:30.047464 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55"] Oct 31 00:55:30.096893 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:30.096856 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b86695-6rx55"] Oct 31 00:55:32.042439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.042399 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfb90df-62a9-4acd-afc1-3df560652008" path="/var/lib/kubelet/pods/5bfb90df-62a9-4acd-afc1-3df560652008/volumes" Oct 31 00:55:32.741618 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.741587 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb"] Oct 31 00:55:32.746936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.746914 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.800069 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.800033 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c6b3b9\"/\"default-dockercfg-cf2g6\"" Oct 31 00:55:32.837950 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.837916 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:32.849646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.849624 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c6b3b9\"/\"openshift-service-ca.crt\"" Oct 31 00:55:32.849646 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.849633 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c6b3b9\"/\"kube-root-ca.crt\"" Oct 31 00:55:32.852075 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.852055 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb"] Oct 31 00:55:32.888769 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.888725 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.888971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.888800 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bvq\" (UniqueName: \"kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.888971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.888869 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.888971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.888922 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.888971 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.888953 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.889147 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.889007 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.900236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.900134 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c6b3b9\"/\"auth-test-model-c6b3b9-kserve-self-signed-certs\"" Oct 31 00:55:32.990140 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990109 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.990396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990162 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.990396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990262 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.990396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990352 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.990575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990429 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bvq\" (UniqueName: \"kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.990575 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990531 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.990942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.990646 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.991068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.991048 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.991369 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.991349 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.992787 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.992746 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:32.993031 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:32.993011 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:33.054472 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:33.054437 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bvq\" (UniqueName: \"kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq\") pod \"auth-test-model-c6b3b9-kserve-794858df5c-2fscb\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:33.062164 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:33.062110 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:33.479561 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:33.479534 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb"] Oct 31 00:55:33.481794 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:55:33.481762 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf34f04_7e4f_453b_b920_2f61002e1cda.slice/crio-cb908bd4b7d090d0fd70915a0cfb8e740bc9d15c5825559da757e1930cda1019 WatchSource:0}: Error finding container cb908bd4b7d090d0fd70915a0cfb8e740bc9d15c5825559da757e1930cda1019: Status 404 returned error can't find the container with id cb908bd4b7d090d0fd70915a0cfb8e740bc9d15c5825559da757e1930cda1019 Oct 31 00:55:34.148413 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:34.148302 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.343662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:34.343621 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" event={"ID":"baf34f04-7e4f-453b-b920-2f61002e1cda","Type":"ContainerStarted","Data":"6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2"} Oct 31 00:55:34.343662 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:34.343667 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" event={"ID":"baf34f04-7e4f-453b-b920-2f61002e1cda","Type":"ContainerStarted","Data":"cb908bd4b7d090d0fd70915a0cfb8e740bc9d15c5825559da757e1930cda1019"} Oct 31 00:55:34.462072 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:34.461971 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" podStartSLOduration=2.461954689 podStartE2EDuration="2.461954689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:34.459096807 +0000 UTC m=+3114.552387007" watchObservedRunningTime="2026-10-31 00:55:34.461954689 +0000 UTC m=+3114.555244875" Oct 31 00:55:35.062879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:35.062845 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:36.462424 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:36.462389 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:38.916690 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:38.916618 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:41.461080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:41.461039 5943 scope.go:122] "RemoveContainer" containerID="d1614d36de8f4a956bc57145c4bbf8b7ecda64aabc561b4820b1ad24d98137b3" Oct 31 00:55:41.471367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:41.471342 5943 scope.go:122] "RemoveContainer" containerID="2cb87f8ec309574f051dacc3b94ee8efc066346ac8c3801caa9503a2e62122fa" Oct 31 00:55:41.480780 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:41.480746 5943 scope.go:122] "RemoveContainer" containerID="84414f09ae8231d366ec1ec40105a587019e818bb8fdf6e8dbb40e9d720926b4" Oct 31 00:55:41.492377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:41.492349 5943 scope.go:122] "RemoveContainer" containerID="d1d72b746a21fd00d9159902840fd7dc57512e523a1b573e33032ee65fd16c23" Oct 31 00:55:43.063230 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:43.063164 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:43.075962 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:43.075927 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:43.413150 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:43.413070 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 00:55:43.657911 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:43.657876 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:48.443507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:48.443465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.425883 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:51.425836 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn"] Oct 31 00:55:51.426329 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:51.426110 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" containerID="cri-o://a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96" gracePeriod=30 Oct 31 00:55:53.170560 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:53.170527 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:56.107311 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.107269 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:56.644609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.644575 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:55:56.690985 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.690921 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:55:56.802202 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802144 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802266 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802307 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802348 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802384 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802379 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802412 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802441 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802557 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802602 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802588 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert\") pod \"060262bf-3ba9-4455-ab8a-d05061a833bf\" (UID: \"060262bf-3ba9-4455-ab8a-d05061a833bf\") " Oct 31 00:55:56.802818 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802661 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.802937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802794 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.802937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802894 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.802937 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.802882 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.803151 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.803132 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:56.803316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.803294 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.803316 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.803317 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.803489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.803332 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.803489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.803345 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.803489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.803357 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060262bf-3ba9-4455-ab8a-d05061a833bf-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.804966 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.804927 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "kube-api-access-gxh88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:56.805348 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.805323 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:56.805434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.805392 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:56.805434 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.805412 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo" pod "060262bf-3ba9-4455-ab8a-d05061a833bf" (UID: "060262bf-3ba9-4455-ab8a-d05061a833bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:56.904921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.904879 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.904921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.904915 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxh88\" (UniqueName: \"kubernetes.io/projected/060262bf-3ba9-4455-ab8a-d05061a833bf-kube-api-access-gxh88\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.904921 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.904925 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:56.905226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:56.904935 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060262bf-3ba9-4455-ab8a-d05061a833bf-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.477149 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:55:57.477108 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:55:57.494439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.494398 5943 generic.go:356] "Generic (PLEG): container finished" podID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerID="a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96" exitCode=0 Oct 31 00:55:57.494632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.494452 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" Oct 31 00:55:57.494632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.494461 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" event={"ID":"060262bf-3ba9-4455-ab8a-d05061a833bf","Type":"ContainerDied","Data":"a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96"} Oct 31 00:55:57.494632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.494526 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn" event={"ID":"060262bf-3ba9-4455-ab8a-d05061a833bf","Type":"ContainerDied","Data":"d689ae0d74139144f858fc867613b0ab6a72f1166f02c1de60476e01cfae4fbe"} Oct 31 00:55:57.494632 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.494555 5943 scope.go:122] "RemoveContainer" containerID="a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96" Oct 31 00:55:57.505825 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.505807 5943 scope.go:122] "RemoveContainer" containerID="a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96" Oct 31 00:55:57.506120 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:55:57.506100 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96\": container with ID starting with a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96 not found: ID does not exist" containerID="a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96" Oct 31 00:55:57.506212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.506128 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96"} err="failed to get container status \"a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96\": rpc error: code = NotFound desc = could not find container \"a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96\": container with ID starting with a16bb1d36e8a8bd36172c1f894c3e0ef8c845b1d1f3ffef84aad4ad43170bf96 not found: ID does not exist" Oct 31 00:55:57.660261 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.660224 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn"] Oct 31 00:55:57.709377 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:57.709334 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-dxrwn"] Oct 31 00:55:58.012859 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:58.012817 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.045978 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:55:58.045939 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" path="/var/lib/kubelet/pods/060262bf-3ba9-4455-ab8a-d05061a833bf/volumes" Oct 31 00:56:00.782401 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:00.782350 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.741049 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:02.741015 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:07.501248 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:07.501209 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:12.281762 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:12.281724 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:13.964271 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:13.964227 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7"] Oct 31 00:56:13.964776 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:13.964502 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" containerID="cri-o://93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e" gracePeriod=30 Oct 31 00:56:14.286761 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:14.286723 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:17.069614 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:17.069573 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:19.222909 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.222881 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:56:19.382367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382334 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382412 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382464 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382572 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382516 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382621 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382660 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382720 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382765 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382750 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382803 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy\") pod \"7345c8aa-6635-4a7f-9b14-079995d6c771\" (UID: \"7345c8aa-6635-4a7f-9b14-079995d6c771\") " Oct 31 00:56:19.382984 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.382966 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:19.383094 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.383053 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:19.383396 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.383345 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:19.383811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.383752 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:19.383811 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.383800 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.384020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.383827 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.384020 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.383746 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:19.385058 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.385029 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:19.385156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.385057 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:19.385156 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.385127 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:19.385284 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.385167 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f" pod "7345c8aa-6635-4a7f-9b14-079995d6c771" (UID: "7345c8aa-6635-4a7f-9b14-079995d6c771"). InnerVolumeSpecName "kube-api-access-ntn4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:19.485236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485201 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.485236 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485234 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7345c8aa-6635-4a7f-9b14-079995d6c771-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.485489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485246 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.485489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485257 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.485489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485265 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7345c8aa-6635-4a7f-9b14-079995d6c771-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.485489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485274 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7345c8aa-6635-4a7f-9b14-079995d6c771-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.485489 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.485283 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntn4f\" (UniqueName: \"kubernetes.io/projected/7345c8aa-6635-4a7f-9b14-079995d6c771-kube-api-access-ntn4f\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.643062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.642978 5943 generic.go:356] "Generic (PLEG): container finished" podID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerID="93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e" exitCode=0 Oct 31 00:56:19.643062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.643028 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" Oct 31 00:56:19.643291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.643058 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" event={"ID":"7345c8aa-6635-4a7f-9b14-079995d6c771","Type":"ContainerDied","Data":"93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e"} Oct 31 00:56:19.643291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.643117 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7" event={"ID":"7345c8aa-6635-4a7f-9b14-079995d6c771","Type":"ContainerDied","Data":"0fb2a925f294a04db035948e54233e89915af54df537f12e9d5ab3efefb9c80e"} Oct 31 00:56:19.643291 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.643140 5943 scope.go:122] "RemoveContainer" containerID="93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e" Oct 31 00:56:19.654055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.654028 5943 scope.go:122] "RemoveContainer" containerID="93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e" Oct 31 00:56:19.654401 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:56:19.654379 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e\": container with ID starting with 93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e not found: ID does not exist" containerID="93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e" Oct 31 00:56:19.654464 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.654412 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e"} err="failed to get container status \"93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e\": rpc error: code = NotFound desc = could not find container \"93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e\": container with ID starting with 93dc5852a183764698ac8d55c30bef80d4b29fd0683979c37a72b0319bbbdc8e not found: ID does not exist" Oct 31 00:56:19.817824 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.817791 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7"] Oct 31 00:56:19.828053 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:19.828024 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-s7nm7"] Oct 31 00:56:20.042068 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:20.042033 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" path="/var/lib/kubelet/pods/7345c8aa-6635-4a7f-9b14-079995d6c771/volumes" Oct 31 00:56:21.835539 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:21.835502 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:26.554953 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:26.554914 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.477844 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:56:27.477798 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:56:31.372050 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:31.372010 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:34.413848 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:34.413805 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:56:36.148675 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:36.148632 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:41.047742 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:41.047706 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:41.531774 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:41.531742 5943 scope.go:122] "RemoveContainer" containerID="34aab1ccce13a84beb0388e4a983ef93dd9387ee3bc331563e0538354a459ff8" Oct 31 00:56:41.541040 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:41.541022 5943 scope.go:122] "RemoveContainer" containerID="3de1410030f72bea15eac2432a72905c70b88ed10c4acc299744b72c42a19b1f" Oct 31 00:56:44.804661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:44.804628 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.076125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:45.076027 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx"] Oct 31 00:56:45.076341 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:45.076310 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" containerID="cri-o://535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25" gracePeriod=30 Oct 31 00:56:45.853077 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:45.853035 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:50.122708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.122684 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:56:50.123129 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.122749 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 31 00:56:50.342936 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.342909 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:56:50.458737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458648 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.458737 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458705 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458741 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458783 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458832 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458874 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458932 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458969 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.458974 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459020 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:50.459070 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459042 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy\") pod \"658a4219-b591-4867-8511-86823dfd9f14\" (UID: \"658a4219-b591-4867-8511-86823dfd9f14\") " Oct 31 00:56:50.459532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459257 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:50.459532 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459454 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:50.459714 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459693 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:50.459803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459718 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.459803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459736 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.459803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459745 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658a4219-b591-4867-8511-86823dfd9f14-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.459803 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.459754 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.461294 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.461267 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:50.461504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.461485 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:50.461807 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.461789 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:50.461871 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.461854 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h" pod "658a4219-b591-4867-8511-86823dfd9f14" (UID: "658a4219-b591-4867-8511-86823dfd9f14"). InnerVolumeSpecName "kube-api-access-8mj5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:50.561360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.561327 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.561360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.561357 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.561360 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.561368 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mj5h\" (UniqueName: \"kubernetes.io/projected/658a4219-b591-4867-8511-86823dfd9f14-kube-api-access-8mj5h\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.561648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.561377 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658a4219-b591-4867-8511-86823dfd9f14-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.561648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.561387 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658a4219-b591-4867-8511-86823dfd9f14-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:50.611841 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.611805 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:50.839661 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.839622 5943 generic.go:356] "Generic (PLEG): container finished" podID="658a4219-b591-4867-8511-86823dfd9f14" containerID="535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25" exitCode=0 Oct 31 00:56:50.839851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.839678 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" Oct 31 00:56:50.839851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.839710 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" event={"ID":"658a4219-b591-4867-8511-86823dfd9f14","Type":"ContainerDied","Data":"535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25"} Oct 31 00:56:50.839851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.839772 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx" event={"ID":"658a4219-b591-4867-8511-86823dfd9f14","Type":"ContainerDied","Data":"90494068bde875403ab306fb94ba4b61d293b6bebe930bdcc3a0e69b49c72c3e"} Oct 31 00:56:50.839851 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.839800 5943 scope.go:122] "RemoveContainer" containerID="535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25" Oct 31 00:56:50.858475 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.858350 5943 scope.go:122] "RemoveContainer" containerID="535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25" Oct 31 00:56:50.859553 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:56:50.859521 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25\": container with ID starting with 535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25 not found: ID does not exist" containerID="535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25" Oct 31 00:56:50.859755 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:50.859724 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25"} err="failed to get container status \"535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25\": rpc error: code = NotFound desc = could not find container \"535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25\": container with ID starting with 535feb4b91ccd0d338fcf750c9bc48ff33b3d1d5eb2525f77fd5655c899e2b25 not found: ID does not exist" Oct 31 00:56:51.004875 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:51.004836 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx"] Oct 31 00:56:51.054354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:51.054316 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-wg8fx"] Oct 31 00:56:52.042461 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:52.042424 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658a4219-b591-4867-8511-86823dfd9f14" path="/var/lib/kubelet/pods/658a4219-b591-4867-8511-86823dfd9f14/volumes" Oct 31 00:56:54.771284 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:54.771250 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.153055 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:55.152946 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.360429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:55.360393 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:56.481626 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:56:56.481597 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:57.476694 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:56:57.476654 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:57:00.139093 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:00.139055 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:00.361321 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:00.361291 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.919829 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:04.919782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:08.918678 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:08.918634 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:09.069634 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:09.069597 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:09.662273 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:09.662236 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:14.545429 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:14.545396 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:19.335504 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:19.335453 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:23.247466 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:23.247427 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:24.164570 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:24.164537 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:27.485595 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:57:27.485552 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:57:27.560644 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:27.560611 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:28.878338 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:28.878301 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:29.217608 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:29.217513 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:57:33.679573 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:33.679533 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:36.986671 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.986635 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp"] Oct 31 00:57:36.987133 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987120 5943 cpu_manager.go:414] "containerMap: removing container" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" Oct 31 00:57:36.987196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987135 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" Oct 31 00:57:36.987196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987159 5943 cpu_manager.go:414] "containerMap: removing container" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" Oct 31 00:57:36.987196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987164 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" Oct 31 00:57:36.987305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987205 5943 cpu_manager.go:414] "containerMap: removing container" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" Oct 31 00:57:36.987305 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987215 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" Oct 31 00:57:36.987367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987333 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="658a4219-b591-4867-8511-86823dfd9f14" containerName="istio-proxy" Oct 31 00:57:36.987367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987344 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="060262bf-3ba9-4455-ab8a-d05061a833bf" containerName="istio-proxy" Oct 31 00:57:36.987367 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.987351 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="7345c8aa-6635-4a7f-9b14-079995d6c771" containerName="istio-proxy" Oct 31 00:57:36.990895 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:36.990879 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.075891 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.075859 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:37.093006 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.092975 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-093220\"/\"default-dockercfg-g46c7\"" Oct 31 00:57:37.093223 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.093035 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-093220\"/\"auth-test-model-093220-kserve-self-signed-certs\"" Oct 31 00:57:37.093223 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.093054 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-093220\"/\"openshift-service-ca.crt\"" Oct 31 00:57:37.093313 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.093246 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-093220\"/\"kube-root-ca.crt\"" Oct 31 00:57:37.095990 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.095969 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp"] Oct 31 00:57:37.132231 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.132195 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.132577 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.132525 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.132802 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.132779 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ccd\" (UniqueName: \"kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.132964 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.132853 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.133038 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.132977 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.133119 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.133096 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.234650 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.234616 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.234864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.234713 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.234864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.234741 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.234864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.234795 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ccd\" (UniqueName: \"kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.234864 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.234838 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.235144 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.234902 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.235397 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.235365 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.235721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.235634 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.235721 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.235758 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.237604 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.237553 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.237700 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.237635 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.296579 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.296542 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ccd\" (UniqueName: \"kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd\") pod \"auth-test-model-093220-kserve-9c4c8ccdf-n2dtp\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.303433 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.303400 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:37.706301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:37.706274 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp"] Oct 31 00:57:37.708236 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:57:37.708208 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca4604a1_40d2_4f6d_a326_d3bd1a45fe4e.slice/crio-86a22d6cc634458a2459694bbeb970d20a3c11e2fb52289079e3d7a920bf6c34 WatchSource:0}: Error finding container 86a22d6cc634458a2459694bbeb970d20a3c11e2fb52289079e3d7a920bf6c34: Status 404 returned error can't find the container with id 86a22d6cc634458a2459694bbeb970d20a3c11e2fb52289079e3d7a920bf6c34 Oct 31 00:57:38.138013 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:38.137983 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" event={"ID":"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e","Type":"ContainerStarted","Data":"3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e"} Oct 31 00:57:38.138482 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:38.138025 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" event={"ID":"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e","Type":"ContainerStarted","Data":"86a22d6cc634458a2459694bbeb970d20a3c11e2fb52289079e3d7a920bf6c34"} Oct 31 00:57:38.185676 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:38.185639 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:38.268708 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:38.268658 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" podStartSLOduration=2.268643777 podStartE2EDuration="2.268643777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:38.264973377 +0000 UTC m=+3238.358263574" watchObservedRunningTime="2026-10-31 00:57:38.268643777 +0000 UTC m=+3238.361933963" Oct 31 00:57:38.304066 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:38.304033 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:38.635978 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:38.635935 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:43.452609 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:43.452573 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:45.147522 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:45.147473 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:46.123480 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:46.123443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:46.984501 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:46.984458 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:47.245421 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:47.245334 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:47.304354 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:47.304322 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:47.317889 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:47.317865 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:48.217235 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:48.217169 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 00:57:48.339887 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:48.339851 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:51.344960 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:51.344840 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:51.560711 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:51.560666 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:53.484275 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:53.484237 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:54.606113 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:54.606078 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:54.785704 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:54.785666 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:56.846109 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:56.846062 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:57.477604 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:57:57.477564 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:57:58.427554 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:57:58.427517 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:03.292901 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:03.292864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:08.258743 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:08.258705 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:13.217562 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:13.217515 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:18.097333 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:18.097293 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:22.996774 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:22.996739 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:25.881657 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:25.881620 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:58:27.476625 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:58:27.476588 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:58:27.796868 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:27.796828 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:32.793605 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:32.793558 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:37.634541 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:37.634499 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:42.703607 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:42.703571 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:47.014592 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:47.014555 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:47.610249 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:47.610211 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:52.649067 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:52.649023 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:57.228101 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:57.228065 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:57.476107 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:58:57.476061 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:58:57.492768 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:58:57.492695 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.240160 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:02.240120 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.436353 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:02.436320 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.732611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:02.732578 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:07.522391 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:07.522349 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:10.558603 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:10.558562 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 00:59:12.622444 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:12.622409 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:16.593802 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.593764 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc"] Oct 31 00:59:16.598060 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.598041 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.686488 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.686448 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvb77\" (UniqueName: \"kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.686667 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.686593 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.686748 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.686681 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.701042 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.701016 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:59:16.701229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.701018 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:16.701229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.701087 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:16.701229 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.701092 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:16.703343 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.703326 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc"] Oct 31 00:59:16.788348 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.788316 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvb77\" (UniqueName: \"kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.788556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.788385 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.788556 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.788422 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.789196 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.789141 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.790959 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.790931 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.859242 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.859145 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvb77\" (UniqueName: \"kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77\") pod \"maas-api-revoke-sub-100e23e9-sv2qc\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:16.911655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:16.911630 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:17.305301 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:17.305278 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc"] Oct 31 00:59:17.307945 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:59:17.307919 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01919f1b_4ca7_4458_b8bd_42eda44f3631.slice/crio-3b7dd3765b4ea6777fbd44c36149e3fa581f39f2812329b59327b76720865a2c WatchSource:0}: Error finding container 3b7dd3765b4ea6777fbd44c36149e3fa581f39f2812329b59327b76720865a2c: Status 404 returned error can't find the container with id 3b7dd3765b4ea6777fbd44c36149e3fa581f39f2812329b59327b76720865a2c Oct 31 00:59:17.520240 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:17.519045 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:17.746917 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:17.746878 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" event={"ID":"01919f1b-4ca7-4458-b8bd-42eda44f3631","Type":"ContainerStarted","Data":"4bbb7a4f5a2f5c29a7e4274069c915e0a962abbc287efa6cd5743ae23d1f0d9e"} Oct 31 00:59:17.747380 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:17.746944 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" event={"ID":"01919f1b-4ca7-4458-b8bd-42eda44f3631","Type":"ContainerStarted","Data":"3b7dd3765b4ea6777fbd44c36149e3fa581f39f2812329b59327b76720865a2c"} Oct 31 00:59:17.864648 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:17.864588 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" podStartSLOduration=1.864573804 podStartE2EDuration="1.864573804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:17.861930093 +0000 UTC m=+3337.955220296" watchObservedRunningTime="2026-10-31 00:59:17.864573804 +0000 UTC m=+3337.957863990" Oct 31 00:59:18.755226 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:18.755170 5943 generic.go:356] "Generic (PLEG): container finished" podID="01919f1b-4ca7-4458-b8bd-42eda44f3631" containerID="4bbb7a4f5a2f5c29a7e4274069c915e0a962abbc287efa6cd5743ae23d1f0d9e" exitCode=0 Oct 31 00:59:18.755622 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:18.755253 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" event={"ID":"01919f1b-4ca7-4458-b8bd-42eda44f3631","Type":"ContainerDied","Data":"4bbb7a4f5a2f5c29a7e4274069c915e0a962abbc287efa6cd5743ae23d1f0d9e"} Oct 31 00:59:19.909166 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:19.909141 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:20.023001 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.022912 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77\" (UniqueName: \"kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77\") pod \"01919f1b-4ca7-4458-b8bd-42eda44f3631\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " Oct 31 00:59:20.023203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.023009 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca\") pod \"01919f1b-4ca7-4458-b8bd-42eda44f3631\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " Oct 31 00:59:20.023203 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.023098 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca\") pod \"01919f1b-4ca7-4458-b8bd-42eda44f3631\" (UID: \"01919f1b-4ca7-4458-b8bd-42eda44f3631\") " Oct 31 00:59:20.023476 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.023451 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca" pod "01919f1b-4ca7-4458-b8bd-42eda44f3631" (UID: "01919f1b-4ca7-4458-b8bd-42eda44f3631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:20.023691 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.023675 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:20.025163 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.025132 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca" pod "01919f1b-4ca7-4458-b8bd-42eda44f3631" (UID: "01919f1b-4ca7-4458-b8bd-42eda44f3631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:20.025290 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.025193 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77" pod "01919f1b-4ca7-4458-b8bd-42eda44f3631" (UID: "01919f1b-4ca7-4458-b8bd-42eda44f3631"). InnerVolumeSpecName "kube-api-access-rvb77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:20.125117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.125083 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvb77\" (UniqueName: \"kubernetes.io/projected/01919f1b-4ca7-4458-b8bd-42eda44f3631-kube-api-access-rvb77\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:20.125117 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.125118 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01919f1b-4ca7-4458-b8bd-42eda44f3631-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:20.768479 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.768453 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" Oct 31 00:59:20.768655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.768534 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc" event={"ID":"01919f1b-4ca7-4458-b8bd-42eda44f3631","Type":"ContainerDied","Data":"3b7dd3765b4ea6777fbd44c36149e3fa581f39f2812329b59327b76720865a2c"} Oct 31 00:59:20.768655 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:20.768590 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7dd3765b4ea6777fbd44c36149e3fa581f39f2812329b59327b76720865a2c" Oct 31 00:59:22.297993 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:22.297948 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:27.180347 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.180306 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:27.341703 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.341675 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv"] Oct 31 00:59:27.342225 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.342210 5943 cpu_manager.go:414] "containerMap: removing container" podUID="01919f1b-4ca7-4458-b8bd-42eda44f3631" containerName="revoke-keys" Oct 31 00:59:27.342277 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.342227 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01919f1b-4ca7-4458-b8bd-42eda44f3631" containerName="revoke-keys" Oct 31 00:59:27.342328 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.342319 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="01919f1b-4ca7-4458-b8bd-42eda44f3631" containerName="revoke-keys" Oct 31 00:59:27.345912 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.345895 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.405507 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.405477 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv"] Oct 31 00:59:27.451652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.451523 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:27.451652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.451560 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:27.451652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.451580 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:27.451652 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.451590 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:59:27.476618 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:59:27.476584 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:59:27.517124 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.517094 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.517294 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.517250 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.517346 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.517332 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7d6z\" (UniqueName: \"kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.619155 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.619122 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.619368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.619210 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.619368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.619268 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7d6z\" (UniqueName: \"kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.619929 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.619902 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.621773 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.621751 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.685212 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.685151 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7d6z\" (UniqueName: \"kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z\") pod \"maas-api-revoke-sub-81b28b9d-bz2hv\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:27.959555 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:27.959523 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:28.335422 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:28.335397 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv"] Oct 31 00:59:28.337549 ip-10-0-143-53 kubenswrapper[5943]: W1031 00:59:28.337524 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301b15f0_4336_4cf2_8c7e_b2cffcbb0a72.slice/crio-34f3385855fd95ce1ec17379767ab9b84c97eb6aadc35163a68bed4f93cad807 WatchSource:0}: Error finding container 34f3385855fd95ce1ec17379767ab9b84c97eb6aadc35163a68bed4f93cad807: Status 404 returned error can't find the container with id 34f3385855fd95ce1ec17379767ab9b84c97eb6aadc35163a68bed4f93cad807 Oct 31 00:59:28.816439 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:28.816398 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" event={"ID":"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72","Type":"ContainerStarted","Data":"bf68d2f597d8935886e23dd6478db7477cf000bbf5c085f1e3642acaeb00a1c4"} Oct 31 00:59:28.816611 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:28.816456 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" event={"ID":"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72","Type":"ContainerStarted","Data":"34f3385855fd95ce1ec17379767ab9b84c97eb6aadc35163a68bed4f93cad807"} Oct 31 00:59:28.932559 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:28.932378 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" podStartSLOduration=1.932362095 podStartE2EDuration="1.932362095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:28.931740667 +0000 UTC m=+3349.025030847" watchObservedRunningTime="2026-10-31 00:59:28.932362095 +0000 UTC m=+3349.025652281" Oct 31 00:59:29.824527 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:29.824492 5943 generic.go:356] "Generic (PLEG): container finished" podID="301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" containerID="bf68d2f597d8935886e23dd6478db7477cf000bbf5c085f1e3642acaeb00a1c4" exitCode=0 Oct 31 00:59:29.824931 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:29.824544 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" event={"ID":"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72","Type":"ContainerDied","Data":"bf68d2f597d8935886e23dd6478db7477cf000bbf5c085f1e3642acaeb00a1c4"} Oct 31 00:59:30.974590 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:30.974558 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:31.164974 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.164890 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca\") pod \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " Oct 31 00:59:31.165125 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.165015 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca\") pod \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " Oct 31 00:59:31.165167 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.165153 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z\" (UniqueName: \"kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z\") pod \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\" (UID: \"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72\") " Oct 31 00:59:31.165426 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.165399 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca" pod "301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" (UID: "301b15f0-4336-4cf2-8c7e-b2cffcbb0a72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:31.165517 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.165502 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:31.167062 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.167040 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca" pod "301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" (UID: "301b15f0-4336-4cf2-8c7e-b2cffcbb0a72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:31.167314 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.167285 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z" pod "301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" (UID: "301b15f0-4336-4cf2-8c7e-b2cffcbb0a72"). InnerVolumeSpecName "kube-api-access-h7d6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:31.266195 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.266155 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7d6z\" (UniqueName: \"kubernetes.io/projected/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-kube-api-access-h7d6z\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:31.266368 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.266217 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:31.838096 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.838068 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" Oct 31 00:59:31.838304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.838107 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv" event={"ID":"301b15f0-4336-4cf2-8c7e-b2cffcbb0a72","Type":"ContainerDied","Data":"34f3385855fd95ce1ec17379767ab9b84c97eb6aadc35163a68bed4f93cad807"} Oct 31 00:59:31.838304 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:31.838148 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f3385855fd95ce1ec17379767ab9b84c97eb6aadc35163a68bed4f93cad807" Oct 31 00:59:32.146448 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:32.146354 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:36.997610 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:36.997569 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:41.155669 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:41.155626 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:41.851416 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:41.851382 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:46.659867 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:46.659817 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:51.459942 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:51.459905 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:56.323080 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:56.323041 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:57.476599 ip-10-0-143-53 kubenswrapper[5943]: E1031 00:59:57.476554 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 00:59:57.586879 ip-10-0-143-53 kubenswrapper[5943]: I1031 00:59:57.586847 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.204656 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.204626 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs"] Oct 31 01:00:00.205306 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.205211 5943 cpu_manager.go:414] "containerMap: removing container" podUID="301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" containerName="revoke-keys" Oct 31 01:00:00.205306 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.205231 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" containerName="revoke-keys" Oct 31 01:00:00.205472 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.205339 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" containerName="revoke-keys" Oct 31 01:00:00.208772 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.208753 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb"] Oct 31 01:00:00.208899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.208869 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.212435 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.212417 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc"] Oct 31 01:00:00.212541 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.212524 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.217335 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.217315 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.253000 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.252974 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r"] Oct 31 01:00:00.256624 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.256598 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246"] Oct 31 01:00:00.256734 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.256687 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.259631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.259611 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh"] Oct 31 01:00:00.259745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.259689 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.262564 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.262543 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr"] Oct 31 01:00:00.262712 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.262656 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.263307 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.263275 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:00:00.265566 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.265545 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv"] Oct 31 01:00:00.265685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.265648 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.268461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.268438 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.300098 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.300077 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:00:00.300222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.300151 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:00:00.302623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.302599 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52"] Oct 31 01:00:00.306150 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.306134 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt"] Oct 31 01:00:00.306282 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.306263 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.309656 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.309629 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb"] Oct 31 01:00:00.309767 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.309719 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.313946 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.313924 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs"] Oct 31 01:00:00.317467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.317450 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r"] Oct 31 01:00:00.322286 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322265 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjdg\" (UniqueName: \"kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.322390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322315 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.322390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322340 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6kq\" (UniqueName: \"kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.322532 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322446 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vfb\" (UniqueName: \"kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.322532 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322518 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwz8\" (UniqueName: \"kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.322655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322563 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.322655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322599 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.322758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322705 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc9h\" (UniqueName: \"kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.322758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322747 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.322876 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322852 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.322934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322900 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.322934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322925 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5nv\" (UniqueName: \"kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.323041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.322974 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrprd\" (UniqueName: \"kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.323041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.323020 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.323041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.323038 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9kt\" (UniqueName: \"kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.323216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.323066 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.353609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.353571 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246"] Oct 31 01:00:00.365688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.365656 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc"] Oct 31 01:00:00.368422 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.368396 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr"] Oct 31 01:00:00.370846 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.370822 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv"] Oct 31 01:00:00.408277 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.408247 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52"] Oct 31 01:00:00.410819 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.410792 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt"] Oct 31 01:00:00.421373 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.421348 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh"] Oct 31 01:00:00.424716 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.424687 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrn2w\" (UniqueName: \"kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.424841 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.424751 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwz8\" (UniqueName: \"kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.424910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.424842 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.424910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.424894 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.425024 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.424940 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5xb\" (UniqueName: \"kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.425024 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.424981 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc9h\" (UniqueName: \"kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.425131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425043 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.425131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425115 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.425270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425150 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.425270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425201 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5nv\" (UniqueName: \"kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.425270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425240 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrprd\" (UniqueName: \"kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.425436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425277 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.425436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425302 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9kt\" (UniqueName: \"kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.425436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425347 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.425436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425424 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425468 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjdg\" (UniqueName: \"kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425532 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425572 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6kq\" (UniqueName: \"kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425618 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425657 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vfb\" (UniqueName: \"kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425703 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.426458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.425971 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.426887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.426652 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.426887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.426714 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.426887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.426773 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.427277 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.427258 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.427715 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.427694 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.427840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.427821 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.488334 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.487708 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwz8\" (UniqueName: \"kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.488544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.488508 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc9h\" (UniqueName: \"kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.494871 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.494836 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vfb\" (UniqueName: \"kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.495082 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.494927 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6kq\" (UniqueName: \"kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.495204 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.495167 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjdg\" (UniqueName: \"kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg\") pod \"maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.514584 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.514555 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrprd\" (UniqueName: \"kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.515170 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.515150 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5nv\" (UniqueName: \"kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.515506 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.515485 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9kt\" (UniqueName: \"kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.526946 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.526916 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.527032 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.526970 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrn2w\" (UniqueName: \"kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.527099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.527042 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5xb\" (UniqueName: \"kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.527140 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.527107 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:00.527248 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.527229 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.527716 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.527698 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.527900 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.527883 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.534204 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.534167 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:00.540112 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.540092 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:00.573092 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.573062 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:00.579619 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.579404 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:00.586788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.586615 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:00.591064 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.590264 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5xb\" (UniqueName: \"kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.597122 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.597092 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:00.602542 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.602107 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:00.637500 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.636101 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:00.655861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.654238 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrn2w\" (UniqueName: \"kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:00.920590 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.920565 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs"] Oct 31 01:00:00.922949 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:00.922920 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a4b9e9_025f_401d_8c9f_339114b56648.slice/crio-9df8059c271b908bdfed2df09e137af051ed4a488408cfa74689f138c2e21159 WatchSource:0}: Error finding container 9df8059c271b908bdfed2df09e137af051ed4a488408cfa74689f138c2e21159: Status 404 returned error can't find the container with id 9df8059c271b908bdfed2df09e137af051ed4a488408cfa74689f138c2e21159 Oct 31 01:00:00.922949 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:00.922940 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:01.021303 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.021261 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerStarted","Data":"9df8059c271b908bdfed2df09e137af051ed4a488408cfa74689f138c2e21159"} Oct 31 01:00:01.072992 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.072966 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r"] Oct 31 01:00:01.109018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.108986 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv"] Oct 31 01:00:01.110553 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:01.110522 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fde808b_b415_424c_aed8_04c63e783420.slice/crio-b76e690397c2ffaa01c18e29869136c957331e7d219ac61985c5c2559dac25df WatchSource:0}: Error finding container b76e690397c2ffaa01c18e29869136c957331e7d219ac61985c5c2559dac25df: Status 404 returned error can't find the container with id b76e690397c2ffaa01c18e29869136c957331e7d219ac61985c5c2559dac25df Oct 31 01:00:01.114071 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.114041 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt"] Oct 31 01:00:01.119659 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.119630 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr"] Oct 31 01:00:01.134718 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.134691 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:01.158217 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.158163 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc"] Oct 31 01:00:01.169942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.169917 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb"] Oct 31 01:00:01.225868 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.225839 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246"] Oct 31 01:00:01.231455 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:01.231422 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded208ff5_c3f8_4c73_b150_0fdacb4fc2d0.slice/crio-bf68eaab43942e891dcce18dfaaae8eafb38b2585cc592961bd54ea270680be3 WatchSource:0}: Error finding container bf68eaab43942e891dcce18dfaaae8eafb38b2585cc592961bd54ea270680be3: Status 404 returned error can't find the container with id bf68eaab43942e891dcce18dfaaae8eafb38b2585cc592961bd54ea270680be3 Oct 31 01:00:01.234398 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:01.234366 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaed22ed_3fb2_449f_bdc0_137f13c27449.slice/crio-de21e242b5528e02b1a06f9f828dafbf2f4ec6fec2cfa4fe68112591881ba936 WatchSource:0}: Error finding container de21e242b5528e02b1a06f9f828dafbf2f4ec6fec2cfa4fe68112591881ba936: Status 404 returned error can't find the container with id de21e242b5528e02b1a06f9f828dafbf2f4ec6fec2cfa4fe68112591881ba936 Oct 31 01:00:01.239965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.239940 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh"] Oct 31 01:00:01.316511 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:01.316485 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52"] Oct 31 01:00:01.332840 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:01.332011 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8396b2d_5018_47e2_be0e_90c84fc42a32.slice/crio-9eb9e1fb0577cbdf69379675366678936c5c4f527beb35d697ba33750fcc584c WatchSource:0}: Error finding container 9eb9e1fb0577cbdf69379675366678936c5c4f527beb35d697ba33750fcc584c: Status 404 returned error can't find the container with id 9eb9e1fb0577cbdf69379675366678936c5c4f527beb35d697ba33750fcc584c Oct 31 01:00:01.334492 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:01.334466 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e5d5f7_93a1_4f90_87b9_1d7da09b767a.slice/crio-6b4e2fae1c367281acc5bc0d12a3bac64543f2ee4193208d3b19d465274b8671 WatchSource:0}: Error finding container 6b4e2fae1c367281acc5bc0d12a3bac64543f2ee4193208d3b19d465274b8671: Status 404 returned error can't find the container with id 6b4e2fae1c367281acc5bc0d12a3bac64543f2ee4193208d3b19d465274b8671 Oct 31 01:00:01.335093 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:01.335069 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16caf51a_9887_417a_816d_3840a6456b25.slice/crio-99ba8f57324e75a1bb7d424985199dde71f3b32b41f80ef2a25306612f0316db WatchSource:0}: Error finding container 99ba8f57324e75a1bb7d424985199dde71f3b32b41f80ef2a25306612f0316db: Status 404 returned error can't find the container with id 99ba8f57324e75a1bb7d424985199dde71f3b32b41f80ef2a25306612f0316db Oct 31 01:00:02.031207 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.030045 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerStarted","Data":"83624ed527f7a061a51ce9004c2c9534f7975758e1f8e89ddfbd9a3ec8c2673b"} Oct 31 01:00:02.044621 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044586 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerStarted","Data":"7d5243b68c6c934d1473d925c36ae060f908170ba2f461565471527f7c8c31a9"} Oct 31 01:00:02.044743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044658 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerStarted","Data":"99ba8f57324e75a1bb7d424985199dde71f3b32b41f80ef2a25306612f0316db"} Oct 31 01:00:02.044743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044685 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerStarted","Data":"e046303dbb1f410b136fb72b597cfdcc4dfbdcd2dc2c1fc00eba54193568d68b"} Oct 31 01:00:02.044743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044708 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerStarted","Data":"b76e690397c2ffaa01c18e29869136c957331e7d219ac61985c5c2559dac25df"} Oct 31 01:00:02.044743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044728 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" event={"ID":"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a","Type":"ContainerStarted","Data":"6b4e2fae1c367281acc5bc0d12a3bac64543f2ee4193208d3b19d465274b8671"} Oct 31 01:00:02.044902 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044752 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" event={"ID":"f8396b2d-5018-47e2-be0e-90c84fc42a32","Type":"ContainerStarted","Data":"9eb9e1fb0577cbdf69379675366678936c5c4f527beb35d697ba33750fcc584c"} Oct 31 01:00:02.044941 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044920 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerStarted","Data":"f447ee8470f8c5b2deeab358859ef96df7d9d68c4042be7ac1c00d723b6e778f"} Oct 31 01:00:02.044992 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.044977 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerStarted","Data":"35548145dad81684b64ef93a0ffc6d54dbeb80d0955cc6d5b45cdfe355bcb10c"} Oct 31 01:00:02.046896 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.046872 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" event={"ID":"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0","Type":"ContainerStarted","Data":"bf68eaab43942e891dcce18dfaaae8eafb38b2585cc592961bd54ea270680be3"} Oct 31 01:00:02.049456 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.049004 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerStarted","Data":"7d68de8f48d9a6ac5af31aa53a936e10355f685a32a06a1de3a00bf3fb0da25f"} Oct 31 01:00:02.049456 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.049052 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerStarted","Data":"de21e242b5528e02b1a06f9f828dafbf2f4ec6fec2cfa4fe68112591881ba936"} Oct 31 01:00:02.051281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.051072 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerStarted","Data":"4e65e6255f1065ec133072ef215b0ee23ecbd31b230c7041618d4abad34dd1fb"} Oct 31 01:00:02.051281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.051118 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerStarted","Data":"601b096dcb1eb4bd56edd07d2e9890c18be71b40816707ecf166f335a54c06ee"} Oct 31 01:00:02.053141 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.053095 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" event={"ID":"35d6b700-c9b3-4b94-9a8b-67c0967a93dc","Type":"ContainerStarted","Data":"1807b5fbe085ac6992e369e78fab112edcbcf38df4161cc5a32070351005f1a3"} Oct 31 01:00:02.268721 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.268660 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" podStartSLOduration=2.268646474 podStartE2EDuration="2.268646474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.149421338 +0000 UTC m=+3382.242711527" watchObservedRunningTime="2026-10-31 01:00:02.268646474 +0000 UTC m=+3382.361936662" Oct 31 01:00:02.269231 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.268759 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" podStartSLOduration=2.268753688 podStartE2EDuration="2.268753688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.267467483 +0000 UTC m=+3382.360757670" watchObservedRunningTime="2026-10-31 01:00:02.268753688 +0000 UTC m=+3382.362043877" Oct 31 01:00:02.385089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.384991 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" podStartSLOduration=2.384978221 podStartE2EDuration="2.384978221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.38285686 +0000 UTC m=+3382.476147047" watchObservedRunningTime="2026-10-31 01:00:02.384978221 +0000 UTC m=+3382.478268443" Oct 31 01:00:02.503064 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.503005 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" podStartSLOduration=2.502986521 podStartE2EDuration="2.502986521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.500262793 +0000 UTC m=+3382.593553001" watchObservedRunningTime="2026-10-31 01:00:02.502986521 +0000 UTC m=+3382.596276708" Oct 31 01:00:02.615633 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.615570 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" podStartSLOduration=2.6155552220000002 podStartE2EDuration="2.615555222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.614147976 +0000 UTC m=+3382.707438200" watchObservedRunningTime="2026-10-31 01:00:02.615555222 +0000 UTC m=+3382.708845408" Oct 31 01:00:02.730575 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:02.730528 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" podStartSLOduration=2.730515409 podStartE2EDuration="2.730515409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.727813719 +0000 UTC m=+3382.821103907" watchObservedRunningTime="2026-10-31 01:00:02.730515409 +0000 UTC m=+3382.823805631" Oct 31 01:00:03.061530 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.061494 5943 generic.go:356] "Generic (PLEG): container finished" podID="16caf51a-9887-417a-816d-3840a6456b25" containerID="7d5243b68c6c934d1473d925c36ae060f908170ba2f461565471527f7c8c31a9" exitCode=60 Oct 31 01:00:03.061730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.061573 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerDied","Data":"7d5243b68c6c934d1473d925c36ae060f908170ba2f461565471527f7c8c31a9"} Oct 31 01:00:03.061892 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.061876 5943 scope.go:122] "RemoveContainer" containerID="7d5243b68c6c934d1473d925c36ae060f908170ba2f461565471527f7c8c31a9" Oct 31 01:00:03.063595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.063569 5943 generic.go:356] "Generic (PLEG): container finished" podID="5fde808b-b415-424c-aed8-04c63e783420" containerID="e046303dbb1f410b136fb72b597cfdcc4dfbdcd2dc2c1fc00eba54193568d68b" exitCode=60 Oct 31 01:00:03.063730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.063646 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerDied","Data":"e046303dbb1f410b136fb72b597cfdcc4dfbdcd2dc2c1fc00eba54193568d68b"} Oct 31 01:00:03.063967 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.063943 5943 scope.go:122] "RemoveContainer" containerID="e046303dbb1f410b136fb72b597cfdcc4dfbdcd2dc2c1fc00eba54193568d68b" Oct 31 01:00:03.065861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.065839 5943 generic.go:356] "Generic (PLEG): container finished" podID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerID="1a939374d6bac35b67d2412c69da0b1442d18dd8bb3119c06419e76077400ce8" exitCode=60 Oct 31 01:00:03.065972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.065948 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" event={"ID":"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a","Type":"ContainerDied","Data":"1a939374d6bac35b67d2412c69da0b1442d18dd8bb3119c06419e76077400ce8"} Oct 31 01:00:03.066449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.066432 5943 scope.go:122] "RemoveContainer" containerID="1a939374d6bac35b67d2412c69da0b1442d18dd8bb3119c06419e76077400ce8" Oct 31 01:00:03.067675 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.067657 5943 generic.go:356] "Generic (PLEG): container finished" podID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerID="0eb28f240411605a0a36abd73887396b4c46e377b2deca4b872167b2d5d8d288" exitCode=60 Oct 31 01:00:03.067751 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.067731 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" event={"ID":"f8396b2d-5018-47e2-be0e-90c84fc42a32","Type":"ContainerDied","Data":"0eb28f240411605a0a36abd73887396b4c46e377b2deca4b872167b2d5d8d288"} Oct 31 01:00:03.068013 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.067997 5943 scope.go:122] "RemoveContainer" containerID="0eb28f240411605a0a36abd73887396b4c46e377b2deca4b872167b2d5d8d288" Oct 31 01:00:03.069695 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.069676 5943 generic.go:356] "Generic (PLEG): container finished" podID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerID="f447ee8470f8c5b2deeab358859ef96df7d9d68c4042be7ac1c00d723b6e778f" exitCode=60 Oct 31 01:00:03.069799 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.069739 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerDied","Data":"f447ee8470f8c5b2deeab358859ef96df7d9d68c4042be7ac1c00d723b6e778f"} Oct 31 01:00:03.070008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.069991 5943 scope.go:122] "RemoveContainer" containerID="f447ee8470f8c5b2deeab358859ef96df7d9d68c4042be7ac1c00d723b6e778f" Oct 31 01:00:03.071234 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.071213 5943 generic.go:356] "Generic (PLEG): container finished" podID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerID="d7c12454a9bc8e7be4e5b52f2a4059147d9258f3994e8efdbef671348bab8737" exitCode=60 Oct 31 01:00:03.071316 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.071293 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" event={"ID":"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0","Type":"ContainerDied","Data":"d7c12454a9bc8e7be4e5b52f2a4059147d9258f3994e8efdbef671348bab8737"} Oct 31 01:00:03.071515 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.071498 5943 scope.go:122] "RemoveContainer" containerID="d7c12454a9bc8e7be4e5b52f2a4059147d9258f3994e8efdbef671348bab8737" Oct 31 01:00:03.073063 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.073041 5943 generic.go:356] "Generic (PLEG): container finished" podID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerID="7d68de8f48d9a6ac5af31aa53a936e10355f685a32a06a1de3a00bf3fb0da25f" exitCode=60 Oct 31 01:00:03.073215 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.073115 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerDied","Data":"7d68de8f48d9a6ac5af31aa53a936e10355f685a32a06a1de3a00bf3fb0da25f"} Oct 31 01:00:03.073501 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.073482 5943 scope.go:122] "RemoveContainer" containerID="7d68de8f48d9a6ac5af31aa53a936e10355f685a32a06a1de3a00bf3fb0da25f" Oct 31 01:00:03.077217 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.076343 5943 generic.go:356] "Generic (PLEG): container finished" podID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerID="4e65e6255f1065ec133072ef215b0ee23ecbd31b230c7041618d4abad34dd1fb" exitCode=60 Oct 31 01:00:03.077217 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.076447 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerDied","Data":"4e65e6255f1065ec133072ef215b0ee23ecbd31b230c7041618d4abad34dd1fb"} Oct 31 01:00:03.077217 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.076615 5943 scope.go:122] "RemoveContainer" containerID="4e65e6255f1065ec133072ef215b0ee23ecbd31b230c7041618d4abad34dd1fb" Oct 31 01:00:03.078687 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.078665 5943 generic.go:356] "Generic (PLEG): container finished" podID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerID="a1d46337b24f03845a2a3c2e52ba1855d10bff3ee8e0dd926cbc40fb5cad7540" exitCode=60 Oct 31 01:00:03.078783 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.078763 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" event={"ID":"35d6b700-c9b3-4b94-9a8b-67c0967a93dc","Type":"ContainerDied","Data":"a1d46337b24f03845a2a3c2e52ba1855d10bff3ee8e0dd926cbc40fb5cad7540"} Oct 31 01:00:03.079528 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.079506 5943 scope.go:122] "RemoveContainer" containerID="a1d46337b24f03845a2a3c2e52ba1855d10bff3ee8e0dd926cbc40fb5cad7540" Oct 31 01:00:03.081450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.081431 5943 generic.go:356] "Generic (PLEG): container finished" podID="94a4b9e9-025f-401d-8c9f-339114b56648" containerID="83624ed527f7a061a51ce9004c2c9534f7975758e1f8e89ddfbd9a3ec8c2673b" exitCode=60 Oct 31 01:00:03.081541 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.081460 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerDied","Data":"83624ed527f7a061a51ce9004c2c9534f7975758e1f8e89ddfbd9a3ec8c2673b"} Oct 31 01:00:03.081704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:03.081689 5943 scope.go:122] "RemoveContainer" containerID="83624ed527f7a061a51ce9004c2c9534f7975758e1f8e89ddfbd9a3ec8c2673b" Oct 31 01:00:04.089686 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.089649 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerStarted","Data":"a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba"} Oct 31 01:00:04.091764 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.091737 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerStarted","Data":"0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d"} Oct 31 01:00:04.093554 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.093531 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerStarted","Data":"c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f"} Oct 31 01:00:04.095490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.095467 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" event={"ID":"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a","Type":"ContainerStarted","Data":"dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360"} Oct 31 01:00:04.097402 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.097375 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" event={"ID":"f8396b2d-5018-47e2-be0e-90c84fc42a32","Type":"ContainerStarted","Data":"ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3"} Oct 31 01:00:04.099331 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.099289 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerStarted","Data":"8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8"} Oct 31 01:00:04.103601 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.103576 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" event={"ID":"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0","Type":"ContainerStarted","Data":"c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d"} Oct 31 01:00:04.106238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.106212 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerStarted","Data":"53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef"} Oct 31 01:00:04.108232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.108207 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerStarted","Data":"6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096"} Oct 31 01:00:04.810947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:04.810889 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" podStartSLOduration=4.81087459 podStartE2EDuration="4.81087459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:04.807081399 +0000 UTC m=+3384.900371600" watchObservedRunningTime="2026-10-31 01:00:04.81087459 +0000 UTC m=+3384.904164777" Oct 31 01:00:05.036879 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.036756 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" podStartSLOduration=5.036736684 podStartE2EDuration="5.036736684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:05.035797795 +0000 UTC m=+3385.129088011" watchObservedRunningTime="2026-10-31 01:00:05.036736684 +0000 UTC m=+3385.130026872" Oct 31 01:00:05.116520 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.116482 5943 generic.go:356] "Generic (PLEG): container finished" podID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerID="ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3" exitCode=60 Oct 31 01:00:05.116965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.116565 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" event={"ID":"f8396b2d-5018-47e2-be0e-90c84fc42a32","Type":"ContainerDied","Data":"ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3"} Oct 31 01:00:05.116965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.116631 5943 scope.go:122] "RemoveContainer" containerID="0eb28f240411605a0a36abd73887396b4c46e377b2deca4b872167b2d5d8d288" Oct 31 01:00:05.116965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.116888 5943 scope.go:122] "RemoveContainer" containerID="ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3" Oct 31 01:00:05.117248 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.117228 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246_odh-ai-gateway-infra(f8396b2d-5018-47e2-be0e-90c84fc42a32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" Oct 31 01:00:05.118527 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.118505 5943 generic.go:356] "Generic (PLEG): container finished" podID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerID="8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8" exitCode=60 Oct 31 01:00:05.118614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.118585 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerDied","Data":"8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8"} Oct 31 01:00:05.118852 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.118838 5943 scope.go:122] "RemoveContainer" containerID="8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8" Oct 31 01:00:05.119150 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.119129 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r_odh-ai-gateway-infra(31d1a279-64d8-4a79-9708-603ce9807bb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" Oct 31 01:00:05.120514 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.120497 5943 generic.go:356] "Generic (PLEG): container finished" podID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerID="c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d" exitCode=60 Oct 31 01:00:05.120616 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.120589 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" event={"ID":"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0","Type":"ContainerDied","Data":"c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d"} Oct 31 01:00:05.120885 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.120855 5943 scope.go:122] "RemoveContainer" containerID="c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d" Oct 31 01:00:05.121194 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.121158 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc_odh-ai-gateway-infra(ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" Oct 31 01:00:05.122682 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.122665 5943 generic.go:356] "Generic (PLEG): container finished" podID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerID="53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef" exitCode=60 Oct 31 01:00:05.122778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.122725 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerDied","Data":"53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef"} Oct 31 01:00:05.122956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.122932 5943 scope.go:122] "RemoveContainer" containerID="53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef" Oct 31 01:00:05.123212 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.123171 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb_odh-ai-gateway-infra(aaed22ed-3fb2-449f-bdc0-137f13c27449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" Oct 31 01:00:05.124623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.124596 5943 generic.go:356] "Generic (PLEG): container finished" podID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerID="6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096" exitCode=60 Oct 31 01:00:05.124712 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.124650 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerDied","Data":"6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096"} Oct 31 01:00:05.124923 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.124905 5943 scope.go:122] "RemoveContainer" containerID="6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096" Oct 31 01:00:05.125161 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.125139 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr_odh-ai-gateway-infra(9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" Oct 31 01:00:05.126745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.126723 5943 generic.go:356] "Generic (PLEG): container finished" podID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerID="854bd3b80b3e4fae55356a9741ec031860cf9917cb58e2bab8d79cbe698840fa" exitCode=60 Oct 31 01:00:05.126842 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.126798 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" event={"ID":"35d6b700-c9b3-4b94-9a8b-67c0967a93dc","Type":"ContainerDied","Data":"854bd3b80b3e4fae55356a9741ec031860cf9917cb58e2bab8d79cbe698840fa"} Oct 31 01:00:05.127004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.126984 5943 scope.go:122] "RemoveContainer" containerID="854bd3b80b3e4fae55356a9741ec031860cf9917cb58e2bab8d79cbe698840fa" Oct 31 01:00:05.127250 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.127231 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt_odh-ai-gateway-infra(35d6b700-c9b3-4b94-9a8b-67c0967a93dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" Oct 31 01:00:05.128960 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.128941 5943 generic.go:356] "Generic (PLEG): container finished" podID="94a4b9e9-025f-401d-8c9f-339114b56648" containerID="a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba" exitCode=60 Oct 31 01:00:05.129073 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.128967 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerDied","Data":"a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba"} Oct 31 01:00:05.129240 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.129224 5943 scope.go:122] "RemoveContainer" containerID="a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba" Oct 31 01:00:05.129521 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.129498 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs_odh-ai-gateway-infra(94a4b9e9-025f-401d-8c9f-339114b56648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" Oct 31 01:00:05.131271 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.131249 5943 generic.go:356] "Generic (PLEG): container finished" podID="16caf51a-9887-417a-816d-3840a6456b25" containerID="0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d" exitCode=60 Oct 31 01:00:05.131372 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.131323 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerDied","Data":"0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d"} Oct 31 01:00:05.131502 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.131487 5943 scope.go:122] "RemoveContainer" containerID="0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d" Oct 31 01:00:05.131697 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.131678 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52_odh-ai-gateway-infra(16caf51a-9887-417a-816d-3840a6456b25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" podUID="16caf51a-9887-417a-816d-3840a6456b25" Oct 31 01:00:05.132562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.132544 5943 scope.go:122] "RemoveContainer" containerID="f447ee8470f8c5b2deeab358859ef96df7d9d68c4042be7ac1c00d723b6e778f" Oct 31 01:00:05.133425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.133404 5943 generic.go:356] "Generic (PLEG): container finished" podID="5fde808b-b415-424c-aed8-04c63e783420" containerID="c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f" exitCode=60 Oct 31 01:00:05.133490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.133462 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerDied","Data":"c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f"} Oct 31 01:00:05.133704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.133687 5943 scope.go:122] "RemoveContainer" containerID="c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f" Oct 31 01:00:05.133937 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.133916 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv_odh-ai-gateway-infra(5fde808b-b415-424c-aed8-04c63e783420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" podUID="5fde808b-b415-424c-aed8-04c63e783420" Oct 31 01:00:05.135458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.135439 5943 generic.go:356] "Generic (PLEG): container finished" podID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerID="dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360" exitCode=60 Oct 31 01:00:05.135547 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.135507 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" event={"ID":"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a","Type":"ContainerDied","Data":"dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360"} Oct 31 01:00:05.135702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.135686 5943 scope.go:122] "RemoveContainer" containerID="dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360" Oct 31 01:00:05.135910 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:05.135890 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh_odh-ai-gateway-infra(f0e5d5f7-93a1-4f90-87b9-1d7da09b767a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" Oct 31 01:00:05.147332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.147314 5943 scope.go:122] "RemoveContainer" containerID="d7c12454a9bc8e7be4e5b52f2a4059147d9258f3994e8efdbef671348bab8737" Oct 31 01:00:05.152685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.152634 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" podStartSLOduration=5.152617059 podStartE2EDuration="5.152617059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:05.149999977 +0000 UTC m=+3385.243290180" watchObservedRunningTime="2026-10-31 01:00:05.152617059 +0000 UTC m=+3385.245907247" Oct 31 01:00:05.161636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.161614 5943 scope.go:122] "RemoveContainer" containerID="7d68de8f48d9a6ac5af31aa53a936e10355f685a32a06a1de3a00bf3fb0da25f" Oct 31 01:00:05.175379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.175352 5943 scope.go:122] "RemoveContainer" containerID="4e65e6255f1065ec133072ef215b0ee23ecbd31b230c7041618d4abad34dd1fb" Oct 31 01:00:05.188407 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.188389 5943 scope.go:122] "RemoveContainer" containerID="a1d46337b24f03845a2a3c2e52ba1855d10bff3ee8e0dd926cbc40fb5cad7540" Oct 31 01:00:05.202829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.202807 5943 scope.go:122] "RemoveContainer" containerID="83624ed527f7a061a51ce9004c2c9534f7975758e1f8e89ddfbd9a3ec8c2673b" Oct 31 01:00:05.215769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.215745 5943 scope.go:122] "RemoveContainer" containerID="7d5243b68c6c934d1473d925c36ae060f908170ba2f461565471527f7c8c31a9" Oct 31 01:00:05.228248 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.228228 5943 scope.go:122] "RemoveContainer" containerID="e046303dbb1f410b136fb72b597cfdcc4dfbdcd2dc2c1fc00eba54193568d68b" Oct 31 01:00:05.242510 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.242374 5943 scope.go:122] "RemoveContainer" containerID="1a939374d6bac35b67d2412c69da0b1442d18dd8bb3119c06419e76077400ce8" Oct 31 01:00:05.870687 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.870648 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:00:05.940014 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:05.939981 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:06.144300 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.144210 5943 scope.go:122] "RemoveContainer" containerID="854bd3b80b3e4fae55356a9741ec031860cf9917cb58e2bab8d79cbe698840fa" Oct 31 01:00:06.144753 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.144576 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt_odh-ai-gateway-infra(35d6b700-c9b3-4b94-9a8b-67c0967a93dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" Oct 31 01:00:06.146228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.146169 5943 scope.go:122] "RemoveContainer" containerID="a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba" Oct 31 01:00:06.146523 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.146503 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs_odh-ai-gateway-infra(94a4b9e9-025f-401d-8c9f-339114b56648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" Oct 31 01:00:06.148109 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.148088 5943 scope.go:122] "RemoveContainer" containerID="0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d" Oct 31 01:00:06.148429 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.148397 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52_odh-ai-gateway-infra(16caf51a-9887-417a-816d-3840a6456b25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" podUID="16caf51a-9887-417a-816d-3840a6456b25" Oct 31 01:00:06.150003 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.149984 5943 scope.go:122] "RemoveContainer" containerID="c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f" Oct 31 01:00:06.150322 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.150292 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv_odh-ai-gateway-infra(5fde808b-b415-424c-aed8-04c63e783420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" podUID="5fde808b-b415-424c-aed8-04c63e783420" Oct 31 01:00:06.151753 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.151737 5943 scope.go:122] "RemoveContainer" containerID="dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360" Oct 31 01:00:06.151991 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.151969 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh_odh-ai-gateway-infra(f0e5d5f7-93a1-4f90-87b9-1d7da09b767a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" Oct 31 01:00:06.153385 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.153370 5943 scope.go:122] "RemoveContainer" containerID="ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3" Oct 31 01:00:06.153599 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.153577 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246_odh-ai-gateway-infra(f8396b2d-5018-47e2-be0e-90c84fc42a32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" Oct 31 01:00:06.156371 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.156351 5943 scope.go:122] "RemoveContainer" containerID="8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8" Oct 31 01:00:06.156587 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.156567 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r_odh-ai-gateway-infra(31d1a279-64d8-4a79-9708-603ce9807bb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" Oct 31 01:00:06.157909 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.157884 5943 scope.go:122] "RemoveContainer" containerID="c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d" Oct 31 01:00:06.158112 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.158092 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc_odh-ai-gateway-infra(ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" Oct 31 01:00:06.161346 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.161328 5943 scope.go:122] "RemoveContainer" containerID="53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef" Oct 31 01:00:06.161525 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.161509 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb_odh-ai-gateway-infra(aaed22ed-3fb2-449f-bdc0-137f13c27449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" Oct 31 01:00:06.162397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:06.162383 5943 scope.go:122] "RemoveContainer" containerID="6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096" Oct 31 01:00:06.162552 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:06.162536 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr_odh-ai-gateway-infra(9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" Oct 31 01:00:10.779756 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:10.779716 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:15.034629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:15.034597 5943 scope.go:122] "RemoveContainer" containerID="ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3" Oct 31 01:00:15.035148 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:15.034749 5943 scope.go:122] "RemoveContainer" containerID="8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8" Oct 31 01:00:15.035148 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:15.034915 5943 scope.go:122] "RemoveContainer" containerID="53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef" Oct 31 01:00:15.563666 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:15.563633 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:16.035336 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.035308 5943 scope.go:122] "RemoveContainer" containerID="c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d" Oct 31 01:00:16.229535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.229504 5943 generic.go:356] "Generic (PLEG): container finished" podID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerID="b2b027bf76eca982a69bb7f2ff05042c69cbea3f91f8f7a84b5aae561ccdcadb" exitCode=60 Oct 31 01:00:16.229714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.229585 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" event={"ID":"f8396b2d-5018-47e2-be0e-90c84fc42a32","Type":"ContainerDied","Data":"b2b027bf76eca982a69bb7f2ff05042c69cbea3f91f8f7a84b5aae561ccdcadb"} Oct 31 01:00:16.229714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.229658 5943 scope.go:122] "RemoveContainer" containerID="ca4441c0fca35fd2dbda3b1d6a4822de79741d63be09ed0927aff51ea35071a3" Oct 31 01:00:16.230129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.230114 5943 scope.go:122] "RemoveContainer" containerID="b2b027bf76eca982a69bb7f2ff05042c69cbea3f91f8f7a84b5aae561ccdcadb" Oct 31 01:00:16.230521 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:16.230496 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246_odh-ai-gateway-infra(f8396b2d-5018-47e2-be0e-90c84fc42a32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" Oct 31 01:00:16.231774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.231743 5943 generic.go:356] "Generic (PLEG): container finished" podID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerID="5d925f18f6940e429b5efd61d48ecb1edc3a237ee7d2a8ffd1dd6a05cae6b6db" exitCode=60 Oct 31 01:00:16.231875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.231814 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerDied","Data":"5d925f18f6940e429b5efd61d48ecb1edc3a237ee7d2a8ffd1dd6a05cae6b6db"} Oct 31 01:00:16.232252 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.232234 5943 scope.go:122] "RemoveContainer" containerID="5d925f18f6940e429b5efd61d48ecb1edc3a237ee7d2a8ffd1dd6a05cae6b6db" Oct 31 01:00:16.232584 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:16.232563 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r_odh-ai-gateway-infra(31d1a279-64d8-4a79-9708-603ce9807bb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" Oct 31 01:00:16.233794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.233775 5943 generic.go:356] "Generic (PLEG): container finished" podID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerID="94ddc3a1091ce7729c2412bf24ead05fa89d97b3ac36b29703082b9962977452" exitCode=60 Oct 31 01:00:16.233890 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.233843 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerDied","Data":"94ddc3a1091ce7729c2412bf24ead05fa89d97b3ac36b29703082b9962977452"} Oct 31 01:00:16.234119 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.234106 5943 scope.go:122] "RemoveContainer" containerID="94ddc3a1091ce7729c2412bf24ead05fa89d97b3ac36b29703082b9962977452" Oct 31 01:00:16.234333 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:16.234317 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb_odh-ai-gateway-infra(aaed22ed-3fb2-449f-bdc0-137f13c27449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" Oct 31 01:00:16.244944 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.244922 5943 scope.go:122] "RemoveContainer" containerID="8be75426d3fe00a58feff264c1f528161683469c0a76941f5e9e52e508a05dc8" Oct 31 01:00:16.258082 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:16.258067 5943 scope.go:122] "RemoveContainer" containerID="53db8e87990b1da56a30b3de133365aea5de7e0ee52e639dedefdc3c1a8495ef" Oct 31 01:00:17.034470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.034443 5943 scope.go:122] "RemoveContainer" containerID="a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba" Oct 31 01:00:17.034685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.034628 5943 scope.go:122] "RemoveContainer" containerID="0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d" Oct 31 01:00:17.246665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.246635 5943 generic.go:356] "Generic (PLEG): container finished" podID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerID="1aa8b9a421c0eefdbb64b6250eeac8f4364cebcec35a5276e20362363efde36b" exitCode=60 Oct 31 01:00:17.247050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.246703 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" event={"ID":"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0","Type":"ContainerDied","Data":"1aa8b9a421c0eefdbb64b6250eeac8f4364cebcec35a5276e20362363efde36b"} Oct 31 01:00:17.247050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.246766 5943 scope.go:122] "RemoveContainer" containerID="c6402e8c91c496753bd98398c04df996c3859e512ae607ce5e2ee2617615299d" Oct 31 01:00:17.247211 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.247193 5943 scope.go:122] "RemoveContainer" containerID="1aa8b9a421c0eefdbb64b6250eeac8f4364cebcec35a5276e20362363efde36b" Oct 31 01:00:17.247587 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:17.247555 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc_odh-ai-gateway-infra(ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" Oct 31 01:00:17.355064 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.355032 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246"] Oct 31 01:00:17.471982 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.471951 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb"] Oct 31 01:00:17.531452 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.531425 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:17.563935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.563903 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca\") pod \"f8396b2d-5018-47e2-be0e-90c84fc42a32\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " Oct 31 01:00:17.564109 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.563989 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt\" (UniqueName: \"kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt\") pod \"f8396b2d-5018-47e2-be0e-90c84fc42a32\" (UID: \"f8396b2d-5018-47e2-be0e-90c84fc42a32\") " Oct 31 01:00:17.564366 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.564343 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca" pod "f8396b2d-5018-47e2-be0e-90c84fc42a32" (UID: "f8396b2d-5018-47e2-be0e-90c84fc42a32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.564529 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.564510 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8396b2d-5018-47e2-be0e-90c84fc42a32-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.566725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.566700 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt" pod "f8396b2d-5018-47e2-be0e-90c84fc42a32" (UID: "f8396b2d-5018-47e2-be0e-90c84fc42a32"). InnerVolumeSpecName "kube-api-access-5v9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.585063 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.585026 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r"] Oct 31 01:00:17.637222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.634834 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:17.672221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.666010 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca\") pod \"aaed22ed-3fb2-449f-bdc0-137f13c27449\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " Oct 31 01:00:17.672221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.666139 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv\" (UniqueName: \"kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv\") pod \"aaed22ed-3fb2-449f-bdc0-137f13c27449\" (UID: \"aaed22ed-3fb2-449f-bdc0-137f13c27449\") " Oct 31 01:00:17.672221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.666829 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9kt\" (UniqueName: \"kubernetes.io/projected/f8396b2d-5018-47e2-be0e-90c84fc42a32-kube-api-access-5v9kt\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.672221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.667519 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca" pod "aaed22ed-3fb2-449f-bdc0-137f13c27449" (UID: "aaed22ed-3fb2-449f-bdc0-137f13c27449"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.672221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.669786 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv" pod "aaed22ed-3fb2-449f-bdc0-137f13c27449" (UID: "aaed22ed-3fb2-449f-bdc0-137f13c27449"). InnerVolumeSpecName "kube-api-access-rp5nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.747388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.747363 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:17.768263 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.768228 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca\") pod \"31d1a279-64d8-4a79-9708-603ce9807bb0\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " Oct 31 01:00:17.768582 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.768557 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd\" (UniqueName: \"kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd\") pod \"31d1a279-64d8-4a79-9708-603ce9807bb0\" (UID: \"31d1a279-64d8-4a79-9708-603ce9807bb0\") " Oct 31 01:00:17.768582 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.768572 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca" pod "31d1a279-64d8-4a79-9708-603ce9807bb0" (UID: "31d1a279-64d8-4a79-9708-603ce9807bb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.769061 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.769040 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed22ed-3fb2-449f-bdc0-137f13c27449-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.769061 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.769061 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5nv\" (UniqueName: \"kubernetes.io/projected/aaed22ed-3fb2-449f-bdc0-137f13c27449-kube-api-access-rp5nv\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.769231 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.769072 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a279-64d8-4a79-9708-603ce9807bb0-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.770986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.770964 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd" pod "31d1a279-64d8-4a79-9708-603ce9807bb0" (UID: "31d1a279-64d8-4a79-9708-603ce9807bb0"). InnerVolumeSpecName "kube-api-access-wrprd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.870342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:17.870305 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrprd\" (UniqueName: \"kubernetes.io/projected/31d1a279-64d8-4a79-9708-603ce9807bb0-kube-api-access-wrprd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.045099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.045071 5943 scope.go:122] "RemoveContainer" containerID="c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f" Oct 31 01:00:18.045866 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.045845 5943 scope.go:122] "RemoveContainer" containerID="dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360" Oct 31 01:00:18.046498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.046479 5943 scope.go:122] "RemoveContainer" containerID="854bd3b80b3e4fae55356a9741ec031860cf9917cb58e2bab8d79cbe698840fa" Oct 31 01:00:18.256743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.256712 5943 generic.go:356] "Generic (PLEG): container finished" podID="16caf51a-9887-417a-816d-3840a6456b25" containerID="df3b47cc252aeebaaf255ed933b5360416207a384b1a9348a20b7a6950791cf3" exitCode=60 Oct 31 01:00:18.257143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.256793 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerDied","Data":"df3b47cc252aeebaaf255ed933b5360416207a384b1a9348a20b7a6950791cf3"} Oct 31 01:00:18.257143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.256859 5943 scope.go:122] "RemoveContainer" containerID="0e3686d54fa7742b8d57ef794cb8796b73115505c118bc228bd08df3e8bdc00d" Oct 31 01:00:18.257324 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.257303 5943 scope.go:122] "RemoveContainer" containerID="df3b47cc252aeebaaf255ed933b5360416207a384b1a9348a20b7a6950791cf3" Oct 31 01:00:18.257679 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:18.257651 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52_odh-ai-gateway-infra(16caf51a-9887-417a-816d-3840a6456b25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" podUID="16caf51a-9887-417a-816d-3840a6456b25" Oct 31 01:00:18.258550 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.258524 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" Oct 31 01:00:18.258637 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.258550 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246" event={"ID":"f8396b2d-5018-47e2-be0e-90c84fc42a32","Type":"ContainerDied","Data":"9eb9e1fb0577cbdf69379675366678936c5c4f527beb35d697ba33750fcc584c"} Oct 31 01:00:18.260065 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.260043 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" event={"ID":"31d1a279-64d8-4a79-9708-603ce9807bb0","Type":"ContainerDied","Data":"35548145dad81684b64ef93a0ffc6d54dbeb80d0955cc6d5b45cdfe355bcb10c"} Oct 31 01:00:18.260156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.260049 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r" Oct 31 01:00:18.263659 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.263640 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" Oct 31 01:00:18.263761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.263686 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb" event={"ID":"aaed22ed-3fb2-449f-bdc0-137f13c27449","Type":"ContainerDied","Data":"de21e242b5528e02b1a06f9f828dafbf2f4ec6fec2cfa4fe68112591881ba936"} Oct 31 01:00:18.266032 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.266011 5943 generic.go:356] "Generic (PLEG): container finished" podID="94a4b9e9-025f-401d-8c9f-339114b56648" containerID="ba566bcb841dabdc9a7b54397f6de7984af01383fed2e19877cf172bc818f407" exitCode=60 Oct 31 01:00:18.266117 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.266039 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerDied","Data":"ba566bcb841dabdc9a7b54397f6de7984af01383fed2e19877cf172bc818f407"} Oct 31 01:00:18.266436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.266417 5943 scope.go:122] "RemoveContainer" containerID="ba566bcb841dabdc9a7b54397f6de7984af01383fed2e19877cf172bc818f407" Oct 31 01:00:18.266692 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:18.266673 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs_odh-ai-gateway-infra(94a4b9e9-025f-401d-8c9f-339114b56648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" Oct 31 01:00:18.274668 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.274650 5943 scope.go:122] "RemoveContainer" containerID="b2b027bf76eca982a69bb7f2ff05042c69cbea3f91f8f7a84b5aae561ccdcadb" Oct 31 01:00:18.284112 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.284091 5943 scope.go:122] "RemoveContainer" containerID="5d925f18f6940e429b5efd61d48ecb1edc3a237ee7d2a8ffd1dd6a05cae6b6db" Oct 31 01:00:18.292823 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.292805 5943 scope.go:122] "RemoveContainer" containerID="94ddc3a1091ce7729c2412bf24ead05fa89d97b3ac36b29703082b9962977452" Oct 31 01:00:18.301282 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.301260 5943 scope.go:122] "RemoveContainer" containerID="a924ef488d17308b8f2be735eda5abb6665c99bb3657702033999a086dd897ba" Oct 31 01:00:18.371626 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.371599 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc"] Oct 31 01:00:18.486746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.486710 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb"] Oct 31 01:00:18.491957 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.491933 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4743f8-29890140-q9xbb"] Oct 31 01:00:18.520913 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.520888 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:18.580161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.580097 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca\") pod \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " Oct 31 01:00:18.580306 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.580289 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h\" (UniqueName: \"kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h\") pod \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\" (UID: \"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0\") " Oct 31 01:00:18.580521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.580497 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca" pod "ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" (UID: "ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.580741 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.580726 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.582349 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.582330 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h" pod "ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" (UID: "ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0"). InnerVolumeSpecName "kube-api-access-rwc9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.602843 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.602816 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r"] Oct 31 01:00:18.612710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.612687 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c6b3b9-29890140-t4h9r"] Oct 31 01:00:18.681743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.681711 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc9h\" (UniqueName: \"kubernetes.io/projected/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0-kube-api-access-rwc9h\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.838645 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.838564 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246"] Oct 31 01:00:18.843428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:18.843397 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-16b984-29890140-ht246"] Oct 31 01:00:19.034431 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.034399 5943 scope.go:122] "RemoveContainer" containerID="6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096" Oct 31 01:00:19.275540 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.275511 5943 generic.go:356] "Generic (PLEG): container finished" podID="5fde808b-b415-424c-aed8-04c63e783420" containerID="a9791b5a5ff9919ad05c5df7b36f431538266e4cbe1f111d05c2bf290a30fdd0" exitCode=60 Oct 31 01:00:19.275956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.275581 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerDied","Data":"a9791b5a5ff9919ad05c5df7b36f431538266e4cbe1f111d05c2bf290a30fdd0"} Oct 31 01:00:19.275956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.275642 5943 scope.go:122] "RemoveContainer" containerID="c3d12ded765556b7c0e91a037db7cf74309626d2adfe0d8b27680987bc38c65f" Oct 31 01:00:19.276093 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.276065 5943 scope.go:122] "RemoveContainer" containerID="a9791b5a5ff9919ad05c5df7b36f431538266e4cbe1f111d05c2bf290a30fdd0" Oct 31 01:00:19.276477 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:19.276448 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv_odh-ai-gateway-infra(5fde808b-b415-424c-aed8-04c63e783420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" podUID="5fde808b-b415-424c-aed8-04c63e783420" Oct 31 01:00:19.277638 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.277618 5943 generic.go:356] "Generic (PLEG): container finished" podID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerID="77c5f3e352bd3fc8b4f0236ba3147362ea4fb710c7812d63031218d0e758a693" exitCode=60 Oct 31 01:00:19.277745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.277685 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" event={"ID":"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a","Type":"ContainerDied","Data":"77c5f3e352bd3fc8b4f0236ba3147362ea4fb710c7812d63031218d0e758a693"} Oct 31 01:00:19.278023 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.278008 5943 scope.go:122] "RemoveContainer" containerID="77c5f3e352bd3fc8b4f0236ba3147362ea4fb710c7812d63031218d0e758a693" Oct 31 01:00:19.278361 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:19.278339 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh_odh-ai-gateway-infra(f0e5d5f7-93a1-4f90-87b9-1d7da09b767a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" Oct 31 01:00:19.280966 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.280944 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" Oct 31 01:00:19.281068 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.280981 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc" event={"ID":"ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0","Type":"ContainerDied","Data":"bf68eaab43942e891dcce18dfaaae8eafb38b2585cc592961bd54ea270680be3"} Oct 31 01:00:19.284067 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.284049 5943 generic.go:356] "Generic (PLEG): container finished" podID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerID="b3f947f401738c58561755d39bd14a9e3ce82feb91558d1259a56f59b2273493" exitCode=60 Oct 31 01:00:19.284151 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.284120 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" event={"ID":"35d6b700-c9b3-4b94-9a8b-67c0967a93dc","Type":"ContainerDied","Data":"b3f947f401738c58561755d39bd14a9e3ce82feb91558d1259a56f59b2273493"} Oct 31 01:00:19.284470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.284452 5943 scope.go:122] "RemoveContainer" containerID="b3f947f401738c58561755d39bd14a9e3ce82feb91558d1259a56f59b2273493" Oct 31 01:00:19.284728 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:19.284709 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt_odh-ai-gateway-infra(35d6b700-c9b3-4b94-9a8b-67c0967a93dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" Oct 31 01:00:19.349170 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.349151 5943 scope.go:122] "RemoveContainer" containerID="dcc8d5da0f1d55865a7033094d547429f0ad2f2da7140938e9959d3719538360" Oct 31 01:00:19.365699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.365661 5943 scope.go:122] "RemoveContainer" containerID="1aa8b9a421c0eefdbb64b6250eeac8f4364cebcec35a5276e20362363efde36b" Oct 31 01:00:19.374687 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.374664 5943 scope.go:122] "RemoveContainer" containerID="854bd3b80b3e4fae55356a9741ec031860cf9917cb58e2bab8d79cbe698840fa" Oct 31 01:00:19.385744 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.385718 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52"] Oct 31 01:00:19.551278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.551248 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:19.598105 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.598073 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca\") pod \"16caf51a-9887-417a-816d-3840a6456b25\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " Oct 31 01:00:19.598333 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.598283 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w\" (UniqueName: \"kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w\") pod \"16caf51a-9887-417a-816d-3840a6456b25\" (UID: \"16caf51a-9887-417a-816d-3840a6456b25\") " Oct 31 01:00:19.598531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.598505 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca" pod "16caf51a-9887-417a-816d-3840a6456b25" (UID: "16caf51a-9887-417a-816d-3840a6456b25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.598708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.598692 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16caf51a-9887-417a-816d-3840a6456b25-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.600549 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.600504 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w" pod "16caf51a-9887-417a-816d-3840a6456b25" (UID: "16caf51a-9887-417a-816d-3840a6456b25"). InnerVolumeSpecName "kube-api-access-xrn2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.624796 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.624765 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc"] Oct 31 01:00:19.630468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.630443 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-bf325c-29890140-qjwzc"] Oct 31 01:00:19.705245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.705014 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrn2w\" (UniqueName: \"kubernetes.io/projected/16caf51a-9887-417a-816d-3840a6456b25-kube-api-access-xrn2w\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.736319 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.736284 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs"] Oct 31 01:00:19.878688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.878661 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:19.907564 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.907541 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca\") pod \"94a4b9e9-025f-401d-8c9f-339114b56648\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " Oct 31 01:00:19.907725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.907711 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8\" (UniqueName: \"kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8\") pod \"94a4b9e9-025f-401d-8c9f-339114b56648\" (UID: \"94a4b9e9-025f-401d-8c9f-339114b56648\") " Oct 31 01:00:19.907891 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.907859 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca" pod "94a4b9e9-025f-401d-8c9f-339114b56648" (UID: "94a4b9e9-025f-401d-8c9f-339114b56648"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.908150 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.908136 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a4b9e9-025f-401d-8c9f-339114b56648-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.909853 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:19.909835 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8" pod "94a4b9e9-025f-401d-8c9f-339114b56648" (UID: "94a4b9e9-025f-401d-8c9f-339114b56648"). InnerVolumeSpecName "kube-api-access-jfwz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.009681 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.009648 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwz8\" (UniqueName: \"kubernetes.io/projected/94a4b9e9-025f-401d-8c9f-339114b56648-kube-api-access-jfwz8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.043458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.043432 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" path="/var/lib/kubelet/pods/31d1a279-64d8-4a79-9708-603ce9807bb0/volumes" Oct 31 01:00:20.043893 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.043877 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" path="/var/lib/kubelet/pods/aaed22ed-3fb2-449f-bdc0-137f13c27449/volumes" Oct 31 01:00:20.044278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.044253 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" path="/var/lib/kubelet/pods/ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0/volumes" Oct 31 01:00:20.044997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.044980 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" path="/var/lib/kubelet/pods/f8396b2d-5018-47e2-be0e-90c84fc42a32/volumes" Oct 31 01:00:20.293526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.293498 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" Oct 31 01:00:20.294009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.293495 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52" event={"ID":"16caf51a-9887-417a-816d-3840a6456b25","Type":"ContainerDied","Data":"99ba8f57324e75a1bb7d424985199dde71f3b32b41f80ef2a25306612f0316db"} Oct 31 01:00:20.294009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.293645 5943 scope.go:122] "RemoveContainer" containerID="df3b47cc252aeebaaf255ed933b5360416207a384b1a9348a20b7a6950791cf3" Oct 31 01:00:20.300463 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.300438 5943 generic.go:356] "Generic (PLEG): container finished" podID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerID="b7a895eb7cfcf55b651a46f6f67e5cb3c48a543716f5efca19f5e83f3866f1f8" exitCode=60 Oct 31 01:00:20.300570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.300505 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerDied","Data":"b7a895eb7cfcf55b651a46f6f67e5cb3c48a543716f5efca19f5e83f3866f1f8"} Oct 31 01:00:20.300903 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.300887 5943 scope.go:122] "RemoveContainer" containerID="b7a895eb7cfcf55b651a46f6f67e5cb3c48a543716f5efca19f5e83f3866f1f8" Oct 31 01:00:20.301208 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:20.301155 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr_odh-ai-gateway-infra(9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" Oct 31 01:00:20.303688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.303668 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" event={"ID":"94a4b9e9-025f-401d-8c9f-339114b56648","Type":"ContainerDied","Data":"9df8059c271b908bdfed2df09e137af051ed4a488408cfa74689f138c2e21159"} Oct 31 01:00:20.303781 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.303709 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs" Oct 31 01:00:20.305103 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.305081 5943 scope.go:122] "RemoveContainer" containerID="6a64aca5dfa13db2dc957f68304ca75374853bd8255c5371f76c1b9ce05ee096" Oct 31 01:00:20.319050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.319027 5943 scope.go:122] "RemoveContainer" containerID="ba566bcb841dabdc9a7b54397f6de7984af01383fed2e19877cf172bc818f407" Oct 31 01:00:20.410241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.410214 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv"] Oct 31 01:00:20.447850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.447818 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:20.572891 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.572867 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:20.580276 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.580250 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs"] Oct 31 01:00:20.585275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.585243 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e1a6c-29890140-xksgs"] Oct 31 01:00:20.616467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.616442 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca\") pod \"5fde808b-b415-424c-aed8-04c63e783420\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " Oct 31 01:00:20.616631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.616529 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq\" (UniqueName: \"kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq\") pod \"5fde808b-b415-424c-aed8-04c63e783420\" (UID: \"5fde808b-b415-424c-aed8-04c63e783420\") " Oct 31 01:00:20.616860 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.616838 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca" pod "5fde808b-b415-424c-aed8-04c63e783420" (UID: "5fde808b-b415-424c-aed8-04c63e783420"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.617050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.617020 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fde808b-b415-424c-aed8-04c63e783420-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.619283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.619257 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq" pod "5fde808b-b415-424c-aed8-04c63e783420" (UID: "5fde808b-b415-424c-aed8-04c63e783420"). InnerVolumeSpecName "kube-api-access-5t6kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.700161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.700129 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52"] Oct 31 01:00:20.708514 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.708487 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890140-68m52"] Oct 31 01:00:20.718282 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.718259 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t6kq\" (UniqueName: \"kubernetes.io/projected/5fde808b-b415-424c-aed8-04c63e783420-kube-api-access-5t6kq\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.771048 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.771016 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh"] Oct 31 01:00:20.875649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.875613 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt"] Oct 31 01:00:20.924259 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:20.924235 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:21.021581 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.021549 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca\") pod \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " Oct 31 01:00:21.021756 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.021739 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg\" (UniqueName: \"kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg\") pod \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\" (UID: \"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a\") " Oct 31 01:00:21.021968 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.021912 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca" pod "f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" (UID: "f0e5d5f7-93a1-4f90-87b9-1d7da09b767a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:21.022214 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.022199 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.024091 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.024070 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg" pod "f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" (UID: "f0e5d5f7-93a1-4f90-87b9-1d7da09b767a"). InnerVolumeSpecName "kube-api-access-bdjdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:21.032454 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.032432 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:21.122904 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.122876 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca\") pod \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " Oct 31 01:00:21.123086 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.123033 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb\" (UniqueName: \"kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb\") pod \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\" (UID: \"35d6b700-c9b3-4b94-9a8b-67c0967a93dc\") " Oct 31 01:00:21.123315 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.123291 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca" pod "35d6b700-c9b3-4b94-9a8b-67c0967a93dc" (UID: "35d6b700-c9b3-4b94-9a8b-67c0967a93dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:21.123578 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.123558 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjdg\" (UniqueName: \"kubernetes.io/projected/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a-kube-api-access-bdjdg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.123651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.123582 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.125392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.125368 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb" pod "35d6b700-c9b3-4b94-9a8b-67c0967a93dc" (UID: "35d6b700-c9b3-4b94-9a8b-67c0967a93dc"). InnerVolumeSpecName "kube-api-access-kg5xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:21.224804 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.224767 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5xb\" (UniqueName: \"kubernetes.io/projected/35d6b700-c9b3-4b94-9a8b-67c0967a93dc-kube-api-access-kg5xb\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.315520 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.315494 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" Oct 31 01:00:21.315997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.315533 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt" event={"ID":"35d6b700-c9b3-4b94-9a8b-67c0967a93dc","Type":"ContainerDied","Data":"1807b5fbe085ac6992e369e78fab112edcbcf38df4161cc5a32070351005f1a3"} Oct 31 01:00:21.315997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.315622 5943 scope.go:122] "RemoveContainer" containerID="b3f947f401738c58561755d39bd14a9e3ce82feb91558d1259a56f59b2273493" Oct 31 01:00:21.319746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.319710 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" event={"ID":"5fde808b-b415-424c-aed8-04c63e783420","Type":"ContainerDied","Data":"b76e690397c2ffaa01c18e29869136c957331e7d219ac61985c5c2559dac25df"} Oct 31 01:00:21.319863 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.319820 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv" Oct 31 01:00:21.325269 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.324760 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" event={"ID":"f0e5d5f7-93a1-4f90-87b9-1d7da09b767a","Type":"ContainerDied","Data":"6b4e2fae1c367281acc5bc0d12a3bac64543f2ee4193208d3b19d465274b8671"} Oct 31 01:00:21.325269 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.324849 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh" Oct 31 01:00:21.328975 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.328950 5943 scope.go:122] "RemoveContainer" containerID="a9791b5a5ff9919ad05c5df7b36f431538266e4cbe1f111d05c2bf290a30fdd0" Oct 31 01:00:21.340137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.340120 5943 scope.go:122] "RemoveContainer" containerID="77c5f3e352bd3fc8b4f0236ba3147362ea4fb710c7812d63031218d0e758a693" Oct 31 01:00:21.476428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.476348 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr"] Oct 31 01:00:21.483664 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.483625 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh"] Oct 31 01:00:21.489545 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.489512 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-869cca-29890140-k5ckh"] Oct 31 01:00:21.597039 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.597007 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt"] Oct 31 01:00:21.601826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.601790 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890140-h96mt"] Oct 31 01:00:21.637939 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.637910 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:21.709322 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.709286 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv"] Oct 31 01:00:21.713951 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.713924 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890140-762pv"] Oct 31 01:00:21.729812 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.729737 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca\") pod \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " Oct 31 01:00:21.729812 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.729804 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb\" (UniqueName: \"kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb\") pod \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\" (UID: \"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261\") " Oct 31 01:00:21.730102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.730082 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca" pod "9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" (UID: "9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:21.730268 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.730248 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.732073 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.732048 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb" pod "9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" (UID: "9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261"). InnerVolumeSpecName "kube-api-access-72vfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:21.810620 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.810586 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-e26187"] Oct 31 01:00:21.811080 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811068 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811082 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811101 5943 cpu_manager.go:414] "containerMap: removing container" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811106 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811113 5943 cpu_manager.go:414] "containerMap: removing container" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811118 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811125 5943 cpu_manager.go:414] "containerMap: removing container" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:21.811132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811130 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811140 5943 cpu_manager.go:414] "containerMap: removing container" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811145 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811153 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811157 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811162 5943 cpu_manager.go:414] "containerMap: removing container" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811167 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811172 5943 cpu_manager.go:414] "containerMap: removing container" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811191 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811207 5943 cpu_manager.go:414] "containerMap: removing container" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811213 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811223 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811229 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811240 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811246 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811255 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811262 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811274 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811281 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811300 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811305 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811318 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811322 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811328 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811332 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811341 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811345 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811351 5943 cpu_manager.go:414] "containerMap: removing container" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811356 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811363 5943 cpu_manager.go:414] "containerMap: removing container" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811367 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811373 5943 cpu_manager.go:414] "containerMap: removing container" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811378 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811386 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.811378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811390 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811400 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811405 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811420 5943 cpu_manager.go:414] "containerMap: removing container" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811424 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811434 5943 cpu_manager.go:414] "containerMap: removing container" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811439 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811447 5943 cpu_manager.go:414] "containerMap: removing container" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811451 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811457 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811461 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811570 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811578 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811586 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811591 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811596 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811601 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811608 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811614 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811620 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811625 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811630 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811635 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811639 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811645 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaed22ed-3fb2-449f-bdc0-137f13c27449" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811651 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811657 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811662 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811667 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811675 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811679 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811686 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8396b2d-5018-47e2-be0e-90c84fc42a32" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811690 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811695 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="16caf51a-9887-417a-816d-3840a6456b25" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811701 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fde808b-b415-424c-aed8-04c63e783420" containerName="cleanup" Oct 31 01:00:21.812400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.811709 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:21.815001 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.814984 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:21.840059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.840017 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vfb\" (UniqueName: \"kubernetes.io/projected/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261-kube-api-access-72vfb\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.868628 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.868285 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\"" Oct 31 01:00:21.870836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.870805 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-e26187"] Oct 31 01:00:21.941739 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:21.941699 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtblv\" (UniqueName: \"kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv\") pod \"test-curl-7498-e26187\" (UID: \"1bd5a98b-9230-4910-bf4a-59fc6637240a\") " pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:22.043209 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.043142 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16caf51a-9887-417a-816d-3840a6456b25" path="/var/lib/kubelet/pods/16caf51a-9887-417a-816d-3840a6456b25/volumes" Oct 31 01:00:22.043388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.043323 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtblv\" (UniqueName: \"kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv\") pod \"test-curl-7498-e26187\" (UID: \"1bd5a98b-9230-4910-bf4a-59fc6637240a\") " pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:22.043620 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.043595 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" path="/var/lib/kubelet/pods/35d6b700-c9b3-4b94-9a8b-67c0967a93dc/volumes" Oct 31 01:00:22.043963 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.043950 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fde808b-b415-424c-aed8-04c63e783420" path="/var/lib/kubelet/pods/5fde808b-b415-424c-aed8-04c63e783420/volumes" Oct 31 01:00:22.044624 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.044610 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" path="/var/lib/kubelet/pods/94a4b9e9-025f-401d-8c9f-339114b56648/volumes" Oct 31 01:00:22.044965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.044952 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" path="/var/lib/kubelet/pods/f0e5d5f7-93a1-4f90-87b9-1d7da09b767a/volumes" Oct 31 01:00:22.104272 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.104236 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtblv\" (UniqueName: \"kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv\") pod \"test-curl-7498-e26187\" (UID: \"1bd5a98b-9230-4910-bf4a-59fc6637240a\") " pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:22.128193 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.128156 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:22.338698 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.338598 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" event={"ID":"9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261","Type":"ContainerDied","Data":"601b096dcb1eb4bd56edd07d2e9890c18be71b40816707ecf166f335a54c06ee"} Oct 31 01:00:22.338698 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.338676 5943 scope.go:122] "RemoveContainer" containerID="b7a895eb7cfcf55b651a46f6f67e5cb3c48a543716f5efca19f5e83f3866f1f8" Oct 31 01:00:22.339167 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.338721 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr" Oct 31 01:00:22.505054 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.505018 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr"] Oct 31 01:00:22.510649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.510622 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890140-jn5dr"] Oct 31 01:00:22.513314 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.513293 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-e26187"] Oct 31 01:00:22.557514 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:22.557457 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bd5a98b_9230_4910_bf4a_59fc6637240a.slice/crio-89fd34294001144fef5693d984679d815e832be4dd8e55729d8f1712d7d5f3cf WatchSource:0}: Error finding container 89fd34294001144fef5693d984679d815e832be4dd8e55729d8f1712d7d5f3cf: Status 404 returned error can't find the container with id 89fd34294001144fef5693d984679d815e832be4dd8e55729d8f1712d7d5f3cf Oct 31 01:00:22.560110 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:22.560089 5943 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:00:23.348113 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:23.348083 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-e26187" event={"ID":"1bd5a98b-9230-4910-bf4a-59fc6637240a","Type":"ContainerStarted","Data":"89fd34294001144fef5693d984679d815e832be4dd8e55729d8f1712d7d5f3cf"} Oct 31 01:00:24.041626 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:24.041590 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" path="/var/lib/kubelet/pods/9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261/volumes" Oct 31 01:00:24.356658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:24.356564 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-e26187" event={"ID":"1bd5a98b-9230-4910-bf4a-59fc6637240a","Type":"ContainerStarted","Data":"509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a"} Oct 31 01:00:24.473651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:24.473593 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7498-e26187" podStartSLOduration=2.693253681 podStartE2EDuration="3.473572784s" totalImagesPullingTime="780.319103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:24.472798735 +0000 UTC m=+3404.566088932" watchObservedRunningTime="2026-10-31 01:00:24.473572784 +0000 UTC m=+3404.566862972" Oct 31 01:00:25.181721 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:25.181689 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:25.468700 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:25.468619 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:25.653978 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:25.653943 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:26.117987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.117885 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-e26187"] Oct 31 01:00:26.121799 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.121771 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-e26187"] Oct 31 01:00:26.190651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.190611 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4"] Oct 31 01:00:26.191434 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191403 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:26.191434 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191430 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:26.191538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191500 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" containerName="test-curl-7498-e26187" Oct 31 01:00:26.191538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191510 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" containerName="test-curl-7498-e26187" Oct 31 01:00:26.191538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191531 5943 cpu_manager.go:414] "containerMap: removing container" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:26.191635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191539 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:26.191635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191554 5943 cpu_manager.go:414] "containerMap: removing container" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:26.191635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191561 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:26.191635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191579 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:26.191635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191586 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e5d5f7-93a1-4f90-87b9-1d7da09b767a" containerName="cleanup" Oct 31 01:00:26.191777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191718 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d6b700-c9b3-4b94-9a8b-67c0967a93dc" containerName="cleanup" Oct 31 01:00:26.191777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191733 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed208ff5-c3f8-4c73-b150-0fdacb4fc2d0" containerName="cleanup" Oct 31 01:00:26.191777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191742 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" containerName="test-curl-7498-e26187" Oct 31 01:00:26.191777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191756 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d1a279-64d8-4a79-9708-603ce9807bb0" containerName="cleanup" Oct 31 01:00:26.191777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191765 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a4b9e9-025f-401d-8c9f-339114b56648" containerName="cleanup" Oct 31 01:00:26.191777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.191775 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8d8a2-b9ac-4df2-b5c9-e5b462bc3261" containerName="cleanup" Oct 31 01:00:26.196089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.196062 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.273203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.272505 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e8808d-openshift-default-dockercfg-m6h9l\"" Oct 31 01:00:26.287304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.287069 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297418 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297562 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297617 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297683 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66gv\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297726 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297790 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297834 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297870 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.298239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.297926 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.306024 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.305986 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4"] Oct 31 01:00:26.378613 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.378511 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7498-e26187" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" containerName="test-curl-7498-e26187" containerID="cri-o://509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a" gracePeriod=2 Oct 31 01:00:26.398721 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398681 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.398910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398748 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.398910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398774 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.398910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398810 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66gv\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.398910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398842 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.398910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398880 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.398910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398909 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399368 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.398955 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399368 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.399005 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399368 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.399288 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399368 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.399335 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399593 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.399491 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399593 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.399571 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.399593 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.399575 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.401689 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.401664 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.401864 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.401841 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.430013 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:26.429974 5943 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-e26187\" is forbidden: User \"system:node:ip-10-0-143-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-53.us-west-1.compute.internal' and this object" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:26.459836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.459804 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.460006 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.459944 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66gv\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv\") pod \"e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.510629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.510596 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:26.593043 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.593008 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:26.958116 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.958086 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4"] Oct 31 01:00:26.960404 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:26.960374 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46b4d69_1555_473c_916d_7d466222de4a.slice/crio-c515425b9bcb3a4108b1632631732ee96c56ba1096f7f48555accc5973f5a46d WatchSource:0}: Error finding container c515425b9bcb3a4108b1632631732ee96c56ba1096f7f48555accc5973f5a46d: Status 404 returned error can't find the container with id c515425b9bcb3a4108b1632631732ee96c56ba1096f7f48555accc5973f5a46d Oct 31 01:00:26.963946 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.963896 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:00:26.964076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.963991 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:00:26.964076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:26.964037 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:00:27.387127 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:27.387087 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" event={"ID":"b46b4d69-1555-473c-916d-7d466222de4a","Type":"ContainerStarted","Data":"c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b"} Oct 31 01:00:27.387345 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:27.387154 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" event={"ID":"b46b4d69-1555-473c-916d-7d466222de4a","Type":"ContainerStarted","Data":"c515425b9bcb3a4108b1632631732ee96c56ba1096f7f48555accc5973f5a46d"} Oct 31 01:00:27.477926 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:27.477881 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:00:27.510919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:27.510890 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:27.523038 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:27.522979 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podStartSLOduration=1.5229604270000001 podStartE2EDuration="1.522960427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:27.519264891 +0000 UTC m=+3407.612555079" watchObservedRunningTime="2026-10-31 01:00:27.522960427 +0000 UTC m=+3407.616250616" Oct 31 01:00:27.581545 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:27.581509 5943 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-e26187\" is forbidden: User \"system:node:ip-10-0-143-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-53.us-west-1.compute.internal' and this object" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:27.621214 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:27.620918 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb"] Oct 31 01:00:27.674141 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:27.674036 5943 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-e26187\" is forbidden: User \"system:node:ip-10-0-143-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-53.us-west-1.compute.internal' and this object" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:27.675135 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:27.675107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:28.519962 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:28.519905 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:28.590701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:28.590675 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:28.738608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:28.738524 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv\" (UniqueName: \"kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv\") pod \"1bd5a98b-9230-4910-bf4a-59fc6637240a\" (UID: \"1bd5a98b-9230-4910-bf4a-59fc6637240a\") " Oct 31 01:00:28.740655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:28.740631 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv" pod "1bd5a98b-9230-4910-bf4a-59fc6637240a" (UID: "1bd5a98b-9230-4910-bf4a-59fc6637240a"). InnerVolumeSpecName "kube-api-access-qtblv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:28.840557 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:28.840521 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtblv\" (UniqueName: \"kubernetes.io/projected/1bd5a98b-9230-4910-bf4a-59fc6637240a-kube-api-access-qtblv\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.412755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.412714 5943 generic.go:356] "Generic (PLEG): container finished" podID="1bd5a98b-9230-4910-bf4a-59fc6637240a" containerID="509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a" exitCode=137 Oct 31 01:00:29.412972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.412774 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:29.412972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.412819 5943 scope.go:122] "RemoveContainer" containerID="509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a" Oct 31 01:00:29.413232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.413214 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:29.414278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.414253 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:00:29.423728 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.423706 5943 scope.go:122] "RemoveContainer" containerID="509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a" Oct 31 01:00:29.424027 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:29.424006 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a\": container with ID starting with 509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a not found: ID does not exist" containerID="509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a" Oct 31 01:00:29.424107 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:29.424039 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a"} err="failed to get container status \"509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a\": rpc error: code = NotFound desc = could not find container \"509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a\": container with ID starting with 509ccf7bd4333518477b644591b3455fd688b4ff14cd425dfe94d10fa3065a9a not found: ID does not exist" Oct 31 01:00:29.634745 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:29.634708 5943 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-e26187\" is forbidden: User \"system:node:ip-10-0-143-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-53.us-west-1.compute.internal' and this object" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:30.041835 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:30.041798 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" path="/var/lib/kubelet/pods/1bd5a98b-9230-4910-bf4a-59fc6637240a/volumes" Oct 31 01:00:30.088254 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:30.088213 5943 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-e26187\" is forbidden: User \"system:node:ip-10-0-143-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-53.us-west-1.compute.internal' and this object" podUID="1bd5a98b-9230-4910-bf4a-59fc6637240a" pod="opendatahub/test-curl-7498-e26187" Oct 31 01:00:30.363601 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:30.363522 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:30.763096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:30.763060 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:30.941926 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:30.941888 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:31.099461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:31.099353 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:33.998233 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:33.998165 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:34.169569 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:34.169535 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:34.469331 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:34.469289 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:35.118983 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:35.118941 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:35.509768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:35.509731 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:35.655448 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:35.655384 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:36.120875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:36.120839 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:39.204708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:39.204662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:39.371445 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:39.371398 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:39.533996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:39.533957 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:40.685327 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:40.685234 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:41.617246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:41.617211 5943 scope.go:122] "RemoveContainer" containerID="4711018529bc9c922140481f3b0381d040089280ade1e8ffb19fe13a78820c06" Oct 31 01:00:42.418732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.418693 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:42.618790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.618757 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:42.651772 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.651726 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" podUID="baf34f04-7e4f-453b-b920-2f61002e1cda" containerName="main" containerID="cri-o://6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2" gracePeriod=45 Oct 31 01:00:42.913911 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.913880 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 01:00:42.921267 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.921200 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm\") pod \"baf34f04-7e4f-453b-b920-2f61002e1cda\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " Oct 31 01:00:42.921397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.921374 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache\") pod \"baf34f04-7e4f-453b-b920-2f61002e1cda\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " Oct 31 01:00:42.921466 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.921452 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs\" (UniqueName: \"kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs\") pod \"baf34f04-7e4f-453b-b920-2f61002e1cda\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " Oct 31 01:00:42.921531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.921517 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir\") pod \"baf34f04-7e4f-453b-b920-2f61002e1cda\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " Oct 31 01:00:42.921750 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.921721 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache" pod "baf34f04-7e4f-453b-b920-2f61002e1cda" (UID: "baf34f04-7e4f-453b-b920-2f61002e1cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.921941 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.921862 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir" pod "baf34f04-7e4f-453b-b920-2f61002e1cda" (UID: "baf34f04-7e4f-453b-b920-2f61002e1cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.922141 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.922122 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.922241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.922145 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.923526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.923508 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm" pod "baf34f04-7e4f-453b-b920-2f61002e1cda" (UID: "baf34f04-7e4f-453b-b920-2f61002e1cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.923669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:42.923649 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs" pod "baf34f04-7e4f-453b-b920-2f61002e1cda" (UID: "baf34f04-7e4f-453b-b920-2f61002e1cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:43.023236 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.023200 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home\") pod \"baf34f04-7e4f-453b-b920-2f61002e1cda\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " Oct 31 01:00:43.023477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.023458 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq\" (UniqueName: \"kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq\") pod \"baf34f04-7e4f-453b-b920-2f61002e1cda\" (UID: \"baf34f04-7e4f-453b-b920-2f61002e1cda\") " Oct 31 01:00:43.023568 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.023477 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home" pod "baf34f04-7e4f-453b-b920-2f61002e1cda" (UID: "baf34f04-7e4f-453b-b920-2f61002e1cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:43.023890 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.023873 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf34f04-7e4f-453b-b920-2f61002e1cda-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:43.023955 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.023892 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:43.023955 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.023906 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf34f04-7e4f-453b-b920-2f61002e1cda-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:43.025912 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.025884 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq" pod "baf34f04-7e4f-453b-b920-2f61002e1cda" (UID: "baf34f04-7e4f-453b-b920-2f61002e1cda"). InnerVolumeSpecName "kube-api-access-t7bvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:43.125439 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.125405 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7bvq\" (UniqueName: \"kubernetes.io/projected/baf34f04-7e4f-453b-b920-2f61002e1cda-kube-api-access-t7bvq\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:43.148093 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.148044 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np"] Oct 31 01:00:43.148826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.148808 5943 cpu_manager.go:414] "containerMap: removing container" podUID="baf34f04-7e4f-453b-b920-2f61002e1cda" containerName="main" Oct 31 01:00:43.148890 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.148830 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf34f04-7e4f-453b-b920-2f61002e1cda" containerName="main" Oct 31 01:00:43.149012 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.149001 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf34f04-7e4f-453b-b920-2f61002e1cda" containerName="main" Oct 31 01:00:43.153208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.153163 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.207875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.207750 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-aaa842\"" Oct 31 01:00:43.207875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.207856 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-aaa842-dockercfg-qzvvh\"" Oct 31 01:00:43.211723 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.211696 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np"] Oct 31 01:00:43.227156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.227120 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh\") pod \"payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.227414 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.227395 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.328429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.328380 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.328617 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.328535 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh\") pod \"payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.329159 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.329139 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.389806 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.389765 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh\") pod \"payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.467820 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.467734 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:43.514696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.514662 5943 generic.go:356] "Generic (PLEG): container finished" podID="baf34f04-7e4f-453b-b920-2f61002e1cda" containerID="6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2" exitCode=0 Oct 31 01:00:43.514899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.514741 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" Oct 31 01:00:43.514899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.514749 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" event={"ID":"baf34f04-7e4f-453b-b920-2f61002e1cda","Type":"ContainerDied","Data":"6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2"} Oct 31 01:00:43.514899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.514806 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb" event={"ID":"baf34f04-7e4f-453b-b920-2f61002e1cda","Type":"ContainerDied","Data":"cb908bd4b7d090d0fd70915a0cfb8e740bc9d15c5825559da757e1930cda1019"} Oct 31 01:00:43.514899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.514833 5943 scope.go:122] "RemoveContainer" containerID="6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2" Oct 31 01:00:43.529710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.529689 5943 scope.go:122] "RemoveContainer" containerID="6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2" Oct 31 01:00:43.530115 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:43.530084 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2\": container with ID starting with 6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2 not found: ID does not exist" containerID="6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2" Oct 31 01:00:43.530299 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.530128 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2"} err="failed to get container status \"6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2\": rpc error: code = NotFound desc = could not find container \"6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2\": container with ID starting with 6904ce6ad29201e7e8b4b641af8a83302415f649bc388765b04cc23c0f90b1f2 not found: ID does not exist" Oct 31 01:00:43.680245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.680204 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb"] Oct 31 01:00:43.685612 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.685570 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c6b3b9/auth-test-model-c6b3b9-kserve-794858df5c-2fscb"] Oct 31 01:00:43.847479 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:43.847451 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np"] Oct 31 01:00:43.849137 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:00:43.849110 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2793def7_6e89_412b_b90c_e803f6911f76.slice/crio-7f7794aa6a642dde87ed7f6395e3f81a307b0c7c3de0bddef3c68441c229235e WatchSource:0}: Error finding container 7f7794aa6a642dde87ed7f6395e3f81a307b0c7c3de0bddef3c68441c229235e: Status 404 returned error can't find the container with id 7f7794aa6a642dde87ed7f6395e3f81a307b0c7c3de0bddef3c68441c229235e Oct 31 01:00:44.043694 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:44.043651 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf34f04-7e4f-453b-b920-2f61002e1cda" path="/var/lib/kubelet/pods/baf34f04-7e4f-453b-b920-2f61002e1cda/volumes" Oct 31 01:00:44.523592 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:44.523561 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" event={"ID":"2793def7-6e89-412b-b90c-e803f6911f76","Type":"ContainerStarted","Data":"9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c"} Oct 31 01:00:44.524026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:44.523614 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" event={"ID":"2793def7-6e89-412b-b90c-e803f6911f76","Type":"ContainerStarted","Data":"7f7794aa6a642dde87ed7f6395e3f81a307b0c7c3de0bddef3c68441c229235e"} Oct 31 01:00:44.524026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:44.523674 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:44.641888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:44.641835 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" podStartSLOduration=1.64181139 podStartE2EDuration="1.64181139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:44.639470611 +0000 UTC m=+3424.732760811" watchObservedRunningTime="2026-10-31 01:00:44.64181139 +0000 UTC m=+3424.735101577" Oct 31 01:00:45.813688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:45.813654 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:46.694727 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:46.694690 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:50.882809 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:50.882705 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:51.069724 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:51.069679 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:00:55.532930 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:55.532898 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:00:55.694562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:55.694521 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:55.920210 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:55.920101 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:57.477230 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:00:57.477160 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:00:57.916547 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:00:57.916507 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:00.465856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:00.465806 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:05.255444 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:05.255405 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:09.988124 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:09.988071 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.726338 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:14.726296 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:17.153275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.153238 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw"] Oct 31 01:01:17.157753 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.157729 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.261778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.261747 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:01:17.261778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.261749 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:17.261778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.261748 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:17.262104 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.261839 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:17.264676 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.264653 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw"] Oct 31 01:01:17.266531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.266506 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.266655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.266551 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4qbw\" (UniqueName: \"kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.266718 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.266656 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.367495 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.367461 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.367709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.367512 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4qbw\" (UniqueName: \"kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.367709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.367555 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.368243 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.368221 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.369976 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.369952 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.430392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.430325 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4qbw\" (UniqueName: \"kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw\") pod \"maas-api-revoke-sub-56ed063f-967jw\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.472256 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.472215 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:17.872924 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:17.872895 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw"] Oct 31 01:01:17.878469 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:17.878438 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1c04df_76b0_4854_99ac_fad88427c063.slice/crio-a9ce2bcc757fa5b658779bc17a01b1fbe655affa24db61e48b50d308290a6603 WatchSource:0}: Error finding container a9ce2bcc757fa5b658779bc17a01b1fbe655affa24db61e48b50d308290a6603: Status 404 returned error can't find the container with id a9ce2bcc757fa5b658779bc17a01b1fbe655affa24db61e48b50d308290a6603 Oct 31 01:01:18.738002 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:18.737968 5943 generic.go:356] "Generic (PLEG): container finished" podID="1f1c04df-76b0-4854-99ac-fad88427c063" containerID="dc4c30abb231cca1f949f69d0d59f1cc90073af2c8e9820a406ec49dc9573dba" exitCode=0 Oct 31 01:01:18.738421 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:18.738052 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" event={"ID":"1f1c04df-76b0-4854-99ac-fad88427c063","Type":"ContainerDied","Data":"dc4c30abb231cca1f949f69d0d59f1cc90073af2c8e9820a406ec49dc9573dba"} Oct 31 01:01:18.738421 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:18.738105 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" event={"ID":"1f1c04df-76b0-4854-99ac-fad88427c063","Type":"ContainerStarted","Data":"a9ce2bcc757fa5b658779bc17a01b1fbe655affa24db61e48b50d308290a6603"} Oct 31 01:01:19.551629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:19.551596 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:19.901297 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:19.901267 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:19.999096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:19.999062 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca\") pod \"1f1c04df-76b0-4854-99ac-fad88427c063\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " Oct 31 01:01:19.999314 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:19.999122 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca\") pod \"1f1c04df-76b0-4854-99ac-fad88427c063\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " Oct 31 01:01:19.999314 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:19.999215 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw\" (UniqueName: \"kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw\") pod \"1f1c04df-76b0-4854-99ac-fad88427c063\" (UID: \"1f1c04df-76b0-4854-99ac-fad88427c063\") " Oct 31 01:01:19.999526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:19.999501 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca" pod "1f1c04df-76b0-4854-99ac-fad88427c063" (UID: "1f1c04df-76b0-4854-99ac-fad88427c063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:20.001483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.001456 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca" pod "1f1c04df-76b0-4854-99ac-fad88427c063" (UID: "1f1c04df-76b0-4854-99ac-fad88427c063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:20.001483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.001469 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw" pod "1f1c04df-76b0-4854-99ac-fad88427c063" (UID: "1f1c04df-76b0-4854-99ac-fad88427c063"). InnerVolumeSpecName "kube-api-access-l4qbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:20.100971 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.100889 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:20.100971 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.100919 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1c04df-76b0-4854-99ac-fad88427c063-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:20.100971 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.100929 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4qbw\" (UniqueName: \"kubernetes.io/projected/1f1c04df-76b0-4854-99ac-fad88427c063-kube-api-access-l4qbw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:20.461548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.461473 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs"] Oct 31 01:01:20.461996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.461976 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1f1c04df-76b0-4854-99ac-fad88427c063" containerName="revoke-keys" Oct 31 01:01:20.461996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.461991 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1c04df-76b0-4854-99ac-fad88427c063" containerName="revoke-keys" Oct 31 01:01:20.462133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.462122 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1c04df-76b0-4854-99ac-fad88427c063" containerName="revoke-keys" Oct 31 01:01:20.465643 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.465619 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.523847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.523813 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs"] Oct 31 01:01:20.607482 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.607446 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bhm\" (UniqueName: \"kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.607673 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.607611 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.607673 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.607652 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.708587 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.708549 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.708587 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.708594 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.708827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.708686 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bhm\" (UniqueName: \"kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.709461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.709434 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.711101 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.711081 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.751958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.751877 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" Oct 31 01:01:20.751958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.751920 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw" event={"ID":"1f1c04df-76b0-4854-99ac-fad88427c063","Type":"ContainerDied","Data":"a9ce2bcc757fa5b658779bc17a01b1fbe655affa24db61e48b50d308290a6603"} Oct 31 01:01:20.752133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.751976 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ce2bcc757fa5b658779bc17a01b1fbe655affa24db61e48b50d308290a6603" Oct 31 01:01:20.769386 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.769361 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bhm\" (UniqueName: \"kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm\") pod \"maas-api-revoke-sub-b7ebf049-jjrjs\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:20.789020 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:20.788975 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:21.168118 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:21.168092 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs"] Oct 31 01:01:21.169689 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:21.169657 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7913dd62_b123_407c_9c9a_0edbf1198739.slice/crio-b2ee56fde74dcd1466192c08c5aba9235de1b4269115a43da2907297281db85f WatchSource:0}: Error finding container b2ee56fde74dcd1466192c08c5aba9235de1b4269115a43da2907297281db85f: Status 404 returned error can't find the container with id b2ee56fde74dcd1466192c08c5aba9235de1b4269115a43da2907297281db85f Oct 31 01:01:21.760028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:21.759994 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" event={"ID":"7913dd62-b123-407c-9c9a-0edbf1198739","Type":"ContainerStarted","Data":"9904d3c083c80d95103941855cb02b44652090e569187eb1f51d540aaf33bb37"} Oct 31 01:01:21.760232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:21.760051 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" event={"ID":"7913dd62-b123-407c-9c9a-0edbf1198739","Type":"ContainerStarted","Data":"b2ee56fde74dcd1466192c08c5aba9235de1b4269115a43da2907297281db85f"} Oct 31 01:01:21.875774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:21.875708 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" podStartSLOduration=1.875693185 podStartE2EDuration="1.875693185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:21.874277356 +0000 UTC m=+3461.967567569" watchObservedRunningTime="2026-10-31 01:01:21.875693185 +0000 UTC m=+3461.968983372" Oct 31 01:01:22.768135 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:22.768095 5943 generic.go:356] "Generic (PLEG): container finished" podID="7913dd62-b123-407c-9c9a-0edbf1198739" containerID="9904d3c083c80d95103941855cb02b44652090e569187eb1f51d540aaf33bb37" exitCode=0 Oct 31 01:01:22.768553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:22.768202 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" event={"ID":"7913dd62-b123-407c-9c9a-0edbf1198739","Type":"ContainerDied","Data":"9904d3c083c80d95103941855cb02b44652090e569187eb1f51d540aaf33bb37"} Oct 31 01:01:23.926416 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:23.926384 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:24.054855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.054780 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm\" (UniqueName: \"kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm\") pod \"7913dd62-b123-407c-9c9a-0edbf1198739\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " Oct 31 01:01:24.055034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.054936 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca\") pod \"7913dd62-b123-407c-9c9a-0edbf1198739\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " Oct 31 01:01:24.055034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.054985 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca\") pod \"7913dd62-b123-407c-9c9a-0edbf1198739\" (UID: \"7913dd62-b123-407c-9c9a-0edbf1198739\") " Oct 31 01:01:24.055434 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.055389 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca" pod "7913dd62-b123-407c-9c9a-0edbf1198739" (UID: "7913dd62-b123-407c-9c9a-0edbf1198739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:24.057173 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.057151 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca" pod "7913dd62-b123-407c-9c9a-0edbf1198739" (UID: "7913dd62-b123-407c-9c9a-0edbf1198739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:24.057507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.057489 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm" pod "7913dd62-b123-407c-9c9a-0edbf1198739" (UID: "7913dd62-b123-407c-9c9a-0edbf1198739"). InnerVolumeSpecName "kube-api-access-b8bhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:24.156492 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.156456 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bhm\" (UniqueName: \"kubernetes.io/projected/7913dd62-b123-407c-9c9a-0edbf1198739-kube-api-access-b8bhm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:24.156492 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.156484 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:24.156492 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.156493 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913dd62-b123-407c-9c9a-0edbf1198739-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:24.440166 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.440070 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:24.782007 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.781974 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" event={"ID":"7913dd62-b123-407c-9c9a-0edbf1198739","Type":"ContainerDied","Data":"b2ee56fde74dcd1466192c08c5aba9235de1b4269115a43da2907297281db85f"} Oct 31 01:01:24.782007 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.782004 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs" Oct 31 01:01:24.782245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:24.782021 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ee56fde74dcd1466192c08c5aba9235de1b4269115a43da2907297281db85f" Oct 31 01:01:27.477722 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:01:27.477669 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:01:27.904826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:27.904792 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9"] Oct 31 01:01:27.905414 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:27.905396 5943 cpu_manager.go:414] "containerMap: removing container" podUID="7913dd62-b123-407c-9c9a-0edbf1198739" containerName="revoke-keys" Oct 31 01:01:27.905465 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:27.905416 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7913dd62-b123-407c-9c9a-0edbf1198739" containerName="revoke-keys" Oct 31 01:01:27.905537 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:27.905528 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="7913dd62-b123-407c-9c9a-0edbf1198739" containerName="revoke-keys" Oct 31 01:01:27.909053 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:27.909038 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:27.961346 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:27.961314 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:28.005601 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.005559 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gj49\" (UniqueName: \"kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.005759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.005634 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.005840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.005783 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.011018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.010992 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:28.011164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.011061 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:01:28.014687 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.014665 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9"] Oct 31 01:01:28.061345 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.061306 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:28.107574 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.107535 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gj49\" (UniqueName: \"kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.107767 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.107617 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.107820 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.107760 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.108408 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.108385 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.110058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.110040 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.169121 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.169040 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gj49\" (UniqueName: \"kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49\") pod \"maas-api-revoke-sub-9e78fb6a-gzlv9\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.222550 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.222517 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:28.599171 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.599141 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9"] Oct 31 01:01:28.601797 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:28.601765 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccadd53_f91c_4624_9f43_ca8abc917b38.slice/crio-70f1f50678ed6567e2386a9823aad89a1e410e61debfd5e0a34522378c338a1d WatchSource:0}: Error finding container 70f1f50678ed6567e2386a9823aad89a1e410e61debfd5e0a34522378c338a1d: Status 404 returned error can't find the container with id 70f1f50678ed6567e2386a9823aad89a1e410e61debfd5e0a34522378c338a1d Oct 31 01:01:28.809256 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:28.809216 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" event={"ID":"7ccadd53-f91c-4624-9f43-ca8abc917b38","Type":"ContainerStarted","Data":"70f1f50678ed6567e2386a9823aad89a1e410e61debfd5e0a34522378c338a1d"} Oct 31 01:01:29.249573 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:29.249487 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:29.817559 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:29.817464 5943 generic.go:356] "Generic (PLEG): container finished" podID="7ccadd53-f91c-4624-9f43-ca8abc917b38" containerID="d4b6258655b8e9fd603859e9a039460e7c83e9aeadb175d66c7927cc7ee583ed" exitCode=0 Oct 31 01:01:29.817559 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:29.817546 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" event={"ID":"7ccadd53-f91c-4624-9f43-ca8abc917b38","Type":"ContainerDied","Data":"d4b6258655b8e9fd603859e9a039460e7c83e9aeadb175d66c7927cc7ee583ed"} Oct 31 01:01:30.981567 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:30.981536 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:31.051629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.051588 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca\") pod \"7ccadd53-f91c-4624-9f43-ca8abc917b38\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " Oct 31 01:01:31.051815 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.051736 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca\") pod \"7ccadd53-f91c-4624-9f43-ca8abc917b38\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " Oct 31 01:01:31.051902 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.051843 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49\" (UniqueName: \"kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49\") pod \"7ccadd53-f91c-4624-9f43-ca8abc917b38\" (UID: \"7ccadd53-f91c-4624-9f43-ca8abc917b38\") " Oct 31 01:01:31.052111 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.052088 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca" pod "7ccadd53-f91c-4624-9f43-ca8abc917b38" (UID: "7ccadd53-f91c-4624-9f43-ca8abc917b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:31.052329 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.052311 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:31.053917 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.053895 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca" pod "7ccadd53-f91c-4624-9f43-ca8abc917b38" (UID: "7ccadd53-f91c-4624-9f43-ca8abc917b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:31.054031 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.054014 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49" pod "7ccadd53-f91c-4624-9f43-ca8abc917b38" (UID: "7ccadd53-f91c-4624-9f43-ca8abc917b38"). InnerVolumeSpecName "kube-api-access-7gj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:31.153764 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.153660 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccadd53-f91c-4624-9f43-ca8abc917b38-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:31.153764 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.153701 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gj49\" (UniqueName: \"kubernetes.io/projected/7ccadd53-f91c-4624-9f43-ca8abc917b38-kube-api-access-7gj49\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:31.226063 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.226022 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs"] Oct 31 01:01:31.226660 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.226641 5943 cpu_manager.go:414] "containerMap: removing container" podUID="7ccadd53-f91c-4624-9f43-ca8abc917b38" containerName="revoke-keys" Oct 31 01:01:31.226785 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.226662 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccadd53-f91c-4624-9f43-ca8abc917b38" containerName="revoke-keys" Oct 31 01:01:31.226850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.226825 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccadd53-f91c-4624-9f43-ca8abc917b38" containerName="revoke-keys" Oct 31 01:01:31.230293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.230270 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.288324 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.288291 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs"] Oct 31 01:01:31.355704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.355673 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.355886 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.355828 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.355933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.355900 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95g2\" (UniqueName: \"kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.457559 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.457466 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.457703 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.457564 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95g2\" (UniqueName: \"kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.457703 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.457662 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.458195 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.458155 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.460169 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.460151 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.518742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.518703 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95g2\" (UniqueName: \"kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2\") pod \"maas-api-revoke-sub-98009d16-2vhfs\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.544704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.544671 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:31.830342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.830307 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" event={"ID":"7ccadd53-f91c-4624-9f43-ca8abc917b38","Type":"ContainerDied","Data":"70f1f50678ed6567e2386a9823aad89a1e410e61debfd5e0a34522378c338a1d"} Oct 31 01:01:31.830342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.830328 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9" Oct 31 01:01:31.830557 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.830364 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f1f50678ed6567e2386a9823aad89a1e410e61debfd5e0a34522378c338a1d" Oct 31 01:01:31.922787 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:31.922763 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs"] Oct 31 01:01:31.924399 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:31.924360 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c5a7de_8433_4e55_a66f_8a743f075af1.slice/crio-5769d301f6d89b100657a77db9ed852d9225885a5bea65745c8edc50e4293625 WatchSource:0}: Error finding container 5769d301f6d89b100657a77db9ed852d9225885a5bea65745c8edc50e4293625: Status 404 returned error can't find the container with id 5769d301f6d89b100657a77db9ed852d9225885a5bea65745c8edc50e4293625 Oct 31 01:01:32.838975 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:32.838940 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" event={"ID":"40c5a7de-8433-4e55-a66f-8a743f075af1","Type":"ContainerStarted","Data":"9823fa3067bccbeacaf38cc31ed88a3202b01a36e8f87a7fbe53f8db5c53ff64"} Oct 31 01:01:32.839473 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:32.838995 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" event={"ID":"40c5a7de-8433-4e55-a66f-8a743f075af1","Type":"ContainerStarted","Data":"5769d301f6d89b100657a77db9ed852d9225885a5bea65745c8edc50e4293625"} Oct 31 01:01:33.005415 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:33.005361 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" podStartSLOduration=2.005342731 podStartE2EDuration="2.005342731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:32.954057465 +0000 UTC m=+3473.047347654" watchObservedRunningTime="2026-10-31 01:01:33.005342731 +0000 UTC m=+3473.098632918" Oct 31 01:01:33.852416 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:33.852362 5943 generic.go:356] "Generic (PLEG): container finished" podID="40c5a7de-8433-4e55-a66f-8a743f075af1" containerID="9823fa3067bccbeacaf38cc31ed88a3202b01a36e8f87a7fbe53f8db5c53ff64" exitCode=0 Oct 31 01:01:33.852873 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:33.852566 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" event={"ID":"40c5a7de-8433-4e55-a66f-8a743f075af1","Type":"ContainerDied","Data":"9823fa3067bccbeacaf38cc31ed88a3202b01a36e8f87a7fbe53f8db5c53ff64"} Oct 31 01:01:34.046146 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:34.046116 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:34.556798 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:34.556755 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:01:35.023638 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.023609 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:35.100406 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.100377 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca\") pod \"40c5a7de-8433-4e55-a66f-8a743f075af1\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " Oct 31 01:01:35.100580 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.100489 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2\" (UniqueName: \"kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2\") pod \"40c5a7de-8433-4e55-a66f-8a743f075af1\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " Oct 31 01:01:35.100621 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.100595 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca\") pod \"40c5a7de-8433-4e55-a66f-8a743f075af1\" (UID: \"40c5a7de-8433-4e55-a66f-8a743f075af1\") " Oct 31 01:01:35.100693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.100670 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca" pod "40c5a7de-8433-4e55-a66f-8a743f075af1" (UID: "40c5a7de-8433-4e55-a66f-8a743f075af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:35.100919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.100902 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:35.102855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.102828 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca" pod "40c5a7de-8433-4e55-a66f-8a743f075af1" (UID: "40c5a7de-8433-4e55-a66f-8a743f075af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:35.103028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.103008 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2" pod "40c5a7de-8433-4e55-a66f-8a743f075af1" (UID: "40c5a7de-8433-4e55-a66f-8a743f075af1"). InnerVolumeSpecName "kube-api-access-q95g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:35.201743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.201709 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c5a7de-8433-4e55-a66f-8a743f075af1-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:35.201743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.201737 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q95g2\" (UniqueName: \"kubernetes.io/projected/40c5a7de-8433-4e55-a66f-8a743f075af1-kube-api-access-q95g2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:35.868093 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.868017 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" event={"ID":"40c5a7de-8433-4e55-a66f-8a743f075af1","Type":"ContainerDied","Data":"5769d301f6d89b100657a77db9ed852d9225885a5bea65745c8edc50e4293625"} Oct 31 01:01:35.868093 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.868096 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5769d301f6d89b100657a77db9ed852d9225885a5bea65745c8edc50e4293625" Oct 31 01:01:35.868490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:35.868173 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs" Oct 31 01:01:38.889868 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:38.889831 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:39.126059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.126022 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx"] Oct 31 01:01:39.126684 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.126668 5943 cpu_manager.go:414] "containerMap: removing container" podUID="40c5a7de-8433-4e55-a66f-8a743f075af1" containerName="revoke-keys" Oct 31 01:01:39.126753 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.126686 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c5a7de-8433-4e55-a66f-8a743f075af1" containerName="revoke-keys" Oct 31 01:01:39.126838 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.126827 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c5a7de-8433-4e55-a66f-8a743f075af1" containerName="revoke-keys" Oct 31 01:01:39.130456 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.130439 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.227850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.227762 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:39.227850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.227764 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:39.227850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.227764 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:39.231994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.231976 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:01:39.241553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.241522 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx"] Oct 31 01:01:39.256444 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.256411 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.256642 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.256622 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.256715 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.256663 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq42z\" (UniqueName: \"kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.358301 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.358265 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.358550 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.358446 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.358550 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.358489 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq42z\" (UniqueName: \"kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.359118 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.359090 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.360842 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.360782 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.419866 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.419830 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq42z\" (UniqueName: \"kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z\") pod \"maas-api-revoke-sub-d6d3a743-tq4vx\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.445316 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.445272 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:39.831900 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.831874 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx"] Oct 31 01:01:39.834078 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:39.834050 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7f3c13_5a9b_4cd1_9485_06a41fe960a0.slice/crio-1cc200fe90ff0c853707d2bbdea98e58b58534e25e41df58279d461853f15d20 WatchSource:0}: Error finding container 1cc200fe90ff0c853707d2bbdea98e58b58534e25e41df58279d461853f15d20: Status 404 returned error can't find the container with id 1cc200fe90ff0c853707d2bbdea98e58b58534e25e41df58279d461853f15d20 Oct 31 01:01:39.896477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:39.896445 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" event={"ID":"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0","Type":"ContainerStarted","Data":"1cc200fe90ff0c853707d2bbdea98e58b58534e25e41df58279d461853f15d20"} Oct 31 01:01:40.904422 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:40.904381 5943 generic.go:356] "Generic (PLEG): container finished" podID="ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" containerID="c8eefe9d65430ab2fe7f4fbeb09513746731ce9d191067d96893e387d841eba8" exitCode=0 Oct 31 01:01:40.904845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:40.904469 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" event={"ID":"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0","Type":"ContainerDied","Data":"c8eefe9d65430ab2fe7f4fbeb09513746731ce9d191067d96893e387d841eba8"} Oct 31 01:01:42.066886 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.066853 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:42.195728 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.195692 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca\") pod \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " Oct 31 01:01:42.195922 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.195900 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca\") pod \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " Oct 31 01:01:42.196056 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.196041 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z\" (UniqueName: \"kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z\") pod \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\" (UID: \"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0\") " Oct 31 01:01:42.196140 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.196078 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca" pod "ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" (UID: "ef7f3c13-5a9b-4cd1-9485-06a41fe960a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:42.196556 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.196532 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:42.198289 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.198263 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca" pod "ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" (UID: "ef7f3c13-5a9b-4cd1-9485-06a41fe960a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:42.198416 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.198325 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z" pod "ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" (UID: "ef7f3c13-5a9b-4cd1-9485-06a41fe960a0"). InnerVolumeSpecName "kube-api-access-fq42z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:42.297942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.297902 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:42.297942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.297936 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq42z\" (UniqueName: \"kubernetes.io/projected/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0-kube-api-access-fq42z\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:42.779921 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.779880 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx"] Oct 31 01:01:42.839664 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.839617 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:42.924034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.923981 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" event={"ID":"ef7f3c13-5a9b-4cd1-9485-06a41fe960a0","Type":"ContainerDied","Data":"1cc200fe90ff0c853707d2bbdea98e58b58534e25e41df58279d461853f15d20"} Oct 31 01:01:42.924279 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.924050 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc200fe90ff0c853707d2bbdea98e58b58534e25e41df58279d461853f15d20" Oct 31 01:01:42.924279 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:42.924130 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx" Oct 31 01:01:43.613094 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:43.613056 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:43.772596 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:43.772561 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:43.924492 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:43.924398 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:46.085194 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:46.085119 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:47.917312 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:47.917273 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:48.566296 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.566255 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:48.911804 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.911713 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg"] Oct 31 01:01:48.912483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.912461 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" containerName="revoke-keys" Oct 31 01:01:48.912483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.912481 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" containerName="revoke-keys" Oct 31 01:01:48.912637 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.912604 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" containerName="revoke-keys" Oct 31 01:01:48.916383 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.916365 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.968273 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.968246 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cc6150-openshift-default-dockercfg-2jmzj\"" Oct 31 01:01:48.991760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.991731 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.991947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.991779 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.991947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.991847 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.991947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.991879 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.991947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.991902 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.991947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.991923 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.992340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.992137 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.992340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.992230 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whsk\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.992340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.992279 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:48.999855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:48.999824 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:49.021800 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.021768 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg"] Oct 31 01:01:49.093243 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093167 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093289 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093323 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093408 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093647 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093480 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whsk\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093647 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093562 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093662 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093703 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093735 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.093953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.093801 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.094026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.094005 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.094440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.094321 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.094440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.094389 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.094819 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.094799 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.096566 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.096536 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.096983 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.096960 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.155761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.155731 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whsk\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.156099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.156075 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token\") pod \"e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.230781 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.230684 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:49.300613 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.300314 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:49.669555 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.669526 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg"] Oct 31 01:01:49.671896 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:49.671864 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fd23e6_8479_4811_aa12_3eb2a8521a28.slice/crio-5020ecf3be16c92fc6f45f8d680ef1d08cdb621c451604d11574b4c13f4bebd1 WatchSource:0}: Error finding container 5020ecf3be16c92fc6f45f8d680ef1d08cdb621c451604d11574b4c13f4bebd1: Status 404 returned error can't find the container with id 5020ecf3be16c92fc6f45f8d680ef1d08cdb621c451604d11574b4c13f4bebd1 Oct 31 01:01:49.674861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.674821 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:01:49.674979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.674898 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:01:49.674979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.674931 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:01:49.925936 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.925840 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd"] Oct 31 01:01:49.929957 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.929935 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:49.977042 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.977003 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" event={"ID":"70fd23e6-8479-4811-aa12-3eb2a8521a28","Type":"ContainerStarted","Data":"e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d"} Oct 31 01:01:49.977534 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.977071 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" event={"ID":"70fd23e6-8479-4811-aa12-3eb2a8521a28","Type":"ContainerStarted","Data":"5020ecf3be16c92fc6f45f8d680ef1d08cdb621c451604d11574b4c13f4bebd1"} Oct 31 01:01:49.985037 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.985002 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:49.985262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.985078 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:01:49.985262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.985106 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:49.985755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.985697 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:49.990012 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:49.989979 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd"] Oct 31 01:01:50.006498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.006465 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755cl\" (UniqueName: \"kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.006767 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.006582 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.006767 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.006647 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.107956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.107923 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755cl\" (UniqueName: \"kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.108149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.108028 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.108149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.108112 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.108899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.108875 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.110895 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.110869 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.170086 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.170057 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755cl\" (UniqueName: \"kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl\") pod \"maas-api-revoke-sub-4918fd82-wwbtd\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.170776 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.170725 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podStartSLOduration=2.170705448 podStartE2EDuration="2.170705448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:50.165452628 +0000 UTC m=+3490.258742816" watchObservedRunningTime="2026-10-31 01:01:50.170705448 +0000 UTC m=+3490.263995635" Oct 31 01:01:50.231461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.231369 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:50.237916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.237877 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:50.245075 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.245047 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:50.631449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.631416 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd"] Oct 31 01:01:50.631970 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:01:50.631944 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17889338_f43a_4df9_b0f4_077d82386c39.slice/crio-eb81158bc484cf9bea4d512551bf33f9e7ae423e1c1289c6f80ab8a2463629f5 WatchSource:0}: Error finding container eb81158bc484cf9bea4d512551bf33f9e7ae423e1c1289c6f80ab8a2463629f5: Status 404 returned error can't find the container with id eb81158bc484cf9bea4d512551bf33f9e7ae423e1c1289c6f80ab8a2463629f5 Oct 31 01:01:50.984677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.984629 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" event={"ID":"17889338-f43a-4df9-b0f4-077d82386c39","Type":"ContainerStarted","Data":"275a2e31ebf474b74f9c3e1505f5e5018e1dfb4bc8c87daf8729e8ee622c6194"} Oct 31 01:01:50.985156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.984701 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" event={"ID":"17889338-f43a-4df9-b0f4-077d82386c39","Type":"ContainerStarted","Data":"eb81158bc484cf9bea4d512551bf33f9e7ae423e1c1289c6f80ab8a2463629f5"} Oct 31 01:01:50.985328 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.985307 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:50.986124 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:50.986107 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:01:51.100318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:51.100257 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" podStartSLOduration=2.100239529 podStartE2EDuration="2.100239529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:51.09923895 +0000 UTC m=+3491.192529154" watchObservedRunningTime="2026-10-31 01:01:51.100239529 +0000 UTC m=+3491.193529713" Oct 31 01:01:51.994927 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:51.994831 5943 generic.go:356] "Generic (PLEG): container finished" podID="17889338-f43a-4df9-b0f4-077d82386c39" containerID="275a2e31ebf474b74f9c3e1505f5e5018e1dfb4bc8c87daf8729e8ee622c6194" exitCode=0 Oct 31 01:01:51.995361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:51.994923 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" event={"ID":"17889338-f43a-4df9-b0f4-077d82386c39","Type":"ContainerDied","Data":"275a2e31ebf474b74f9c3e1505f5e5018e1dfb4bc8c87daf8729e8ee622c6194"} Oct 31 01:01:53.064319 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.064197 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:53.170161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.170133 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:53.250152 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.250114 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca\") pod \"17889338-f43a-4df9-b0f4-077d82386c39\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " Oct 31 01:01:53.250390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.250244 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl\" (UniqueName: \"kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl\") pod \"17889338-f43a-4df9-b0f4-077d82386c39\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " Oct 31 01:01:53.250390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.250314 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca\") pod \"17889338-f43a-4df9-b0f4-077d82386c39\" (UID: \"17889338-f43a-4df9-b0f4-077d82386c39\") " Oct 31 01:01:53.250548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.250522 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca" pod "17889338-f43a-4df9-b0f4-077d82386c39" (UID: "17889338-f43a-4df9-b0f4-077d82386c39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:53.250847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.250818 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:53.252539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.252482 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca" pod "17889338-f43a-4df9-b0f4-077d82386c39" (UID: "17889338-f43a-4df9-b0f4-077d82386c39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:53.252636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.252551 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl" pod "17889338-f43a-4df9-b0f4-077d82386c39" (UID: "17889338-f43a-4df9-b0f4-077d82386c39"). InnerVolumeSpecName "kube-api-access-755cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:53.263704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.263674 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:53.352052 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.352012 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755cl\" (UniqueName: \"kubernetes.io/projected/17889338-f43a-4df9-b0f4-077d82386c39-kube-api-access-755cl\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:53.352052 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.352044 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17889338-f43a-4df9-b0f4-077d82386c39-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:53.565376 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:53.565332 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:54.009295 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:54.009269 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" Oct 31 01:01:54.009456 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:54.009297 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd" event={"ID":"17889338-f43a-4df9-b0f4-077d82386c39","Type":"ContainerDied","Data":"eb81158bc484cf9bea4d512551bf33f9e7ae423e1c1289c6f80ab8a2463629f5"} Oct 31 01:01:54.009456 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:54.009357 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb81158bc484cf9bea4d512551bf33f9e7ae423e1c1289c6f80ab8a2463629f5" Oct 31 01:01:57.477322 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:01:57.477278 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:01:57.495621 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:57.495583 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:57.672625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:57.672592 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:57.804726 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:57.804673 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" podUID="780306a8-3607-4c87-8343-f7824cca0b6a" containerName="main" containerID="cri-o://e9d97bfe033e9f850057b58fae952782fb5af286b02c6f85a799d1bbc6271e1f" gracePeriod=45 Oct 31 01:01:58.090850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.090751 5943 generic.go:356] "Generic (PLEG): container finished" podID="780306a8-3607-4c87-8343-f7824cca0b6a" containerID="e9d97bfe033e9f850057b58fae952782fb5af286b02c6f85a799d1bbc6271e1f" exitCode=0 Oct 31 01:01:58.093809 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.093776 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" event={"ID":"780306a8-3607-4c87-8343-f7824cca0b6a","Type":"ContainerDied","Data":"e9d97bfe033e9f850057b58fae952782fb5af286b02c6f85a799d1bbc6271e1f"} Oct 31 01:01:58.143846 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.143809 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 01:01:58.239537 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239505 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir\") pod \"780306a8-3607-4c87-8343-f7824cca0b6a\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " Oct 31 01:01:58.239745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239598 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home\") pod \"780306a8-3607-4c87-8343-f7824cca0b6a\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " Oct 31 01:01:58.239745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239694 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw\" (UniqueName: \"kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw\") pod \"780306a8-3607-4c87-8343-f7824cca0b6a\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " Oct 31 01:01:58.239745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239720 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm\") pod \"780306a8-3607-4c87-8343-f7824cca0b6a\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " Oct 31 01:01:58.239914 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239762 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir" pod "780306a8-3607-4c87-8343-f7824cca0b6a" (UID: "780306a8-3607-4c87-8343-f7824cca0b6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:58.239914 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239770 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache\") pod \"780306a8-3607-4c87-8343-f7824cca0b6a\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " Oct 31 01:01:58.239914 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239875 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs\") pod \"780306a8-3607-4c87-8343-f7824cca0b6a\" (UID: \"780306a8-3607-4c87-8343-f7824cca0b6a\") " Oct 31 01:01:58.239914 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239898 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home" pod "780306a8-3607-4c87-8343-f7824cca0b6a" (UID: "780306a8-3607-4c87-8343-f7824cca0b6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:58.240117 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.239982 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache" pod "780306a8-3607-4c87-8343-f7824cca0b6a" (UID: "780306a8-3607-4c87-8343-f7824cca0b6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:58.240680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.240504 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:58.240680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.240533 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:58.240680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.240549 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:58.242478 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.242452 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw" pod "780306a8-3607-4c87-8343-f7824cca0b6a" (UID: "780306a8-3607-4c87-8343-f7824cca0b6a"). InnerVolumeSpecName "kube-api-access-njfzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:58.242478 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.242459 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm" pod "780306a8-3607-4c87-8343-f7824cca0b6a" (UID: "780306a8-3607-4c87-8343-f7824cca0b6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:58.242673 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.242654 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs" pod "780306a8-3607-4c87-8343-f7824cca0b6a" (UID: "780306a8-3607-4c87-8343-f7824cca0b6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:58.284060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.284020 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:58.342296 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.342201 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njfzw\" (UniqueName: \"kubernetes.io/projected/780306a8-3607-4c87-8343-f7824cca0b6a-kube-api-access-njfzw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:58.342296 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.342235 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780306a8-3607-4c87-8343-f7824cca0b6a-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:58.342296 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.342249 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780306a8-3607-4c87-8343-f7824cca0b6a-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:01:58.738535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.738443 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:58.902252 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:58.902207 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:59.102747 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.102719 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" Oct 31 01:01:59.102747 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.102745 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" event={"ID":"780306a8-3607-4c87-8343-f7824cca0b6a","Type":"ContainerDied","Data":"c962dc4e6b0390384ef32d7283bb7e9d27ffcb4b251bb1a841be630d43f9923f"} Oct 31 01:01:59.103033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.102810 5943 scope.go:122] "RemoveContainer" containerID="e9d97bfe033e9f850057b58fae952782fb5af286b02c6f85a799d1bbc6271e1f" Oct 31 01:01:59.119448 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.119419 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:59.122198 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.122155 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx" podUID="780306a8-3607-4c87-8343-f7824cca0b6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:01:59.267780 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.267744 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx"] Oct 31 01:01:59.271323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:01:59.271290 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f13553/auth-test-model-f13553-kserve-645c49fb45-ckjpx"] Oct 31 01:02:00.042021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:00.041984 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780306a8-3607-4c87-8343-f7824cca0b6a" path="/var/lib/kubelet/pods/780306a8-3607-4c87-8343-f7824cca0b6a/volumes" Oct 31 01:02:03.780783 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:03.780747 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:06.805991 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:06.805949 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w"] Oct 31 01:02:06.829027 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:06.828996 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:07.905562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:07.905506 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:08.538283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:08.538242 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:13.385391 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:13.385340 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:15.455400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:15.455324 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:02:15.455931 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:15.455419 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:02:18.182284 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.182153 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:18.239088 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.239056 5943 generic.go:356] "Generic (PLEG): container finished" podID="b46b4d69-1555-473c-916d-7d466222de4a" containerID="c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b" exitCode=0 Oct 31 01:02:18.239293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.239129 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" event={"ID":"b46b4d69-1555-473c-916d-7d466222de4a","Type":"ContainerDied","Data":"c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b"} Oct 31 01:02:18.239679 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.239663 5943 scope.go:122] "RemoveContainer" containerID="c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b" Oct 31 01:02:18.241406 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.241380 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:02:18.241480 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.241444 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:02:18.241521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:18.241477 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:02:19.261716 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:19.261622 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" event={"ID":"b46b4d69-1555-473c-916d-7d466222de4a","Type":"ContainerStarted","Data":"f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762"} Oct 31 01:02:19.511650 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:19.511611 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:02:19.512002 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:19.511938 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 01:02:19.512116 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:19.512009 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 01:02:20.511592 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:20.511554 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 01:02:20.512021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:20.511634 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 01:02:21.511468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:21.511428 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 01:02:21.511650 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:21.511486 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 01:02:21.827635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:21.827539 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" podUID="b3cda834-3d91-4ae7-a139-daaadce7d091" containerName="main" containerID="cri-o://49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2" gracePeriod=45 Oct 31 01:02:22.184046 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.184023 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 01:02:22.282990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.282953 5943 generic.go:356] "Generic (PLEG): container finished" podID="b3cda834-3d91-4ae7-a139-daaadce7d091" containerID="49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2" exitCode=0 Oct 31 01:02:22.283202 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.283008 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" Oct 31 01:02:22.283202 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.283048 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" event={"ID":"b3cda834-3d91-4ae7-a139-daaadce7d091","Type":"ContainerDied","Data":"49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2"} Oct 31 01:02:22.283202 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.283100 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" event={"ID":"b3cda834-3d91-4ae7-a139-daaadce7d091","Type":"ContainerDied","Data":"c1ae8c8b0def1c5c2878209696c3a9cfaa3150fed82d561eba6396e292cba67b"} Oct 31 01:02:22.283202 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.283117 5943 scope.go:122] "RemoveContainer" containerID="49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2" Oct 31 01:02:22.293888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.293866 5943 scope.go:122] "RemoveContainer" containerID="49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2" Oct 31 01:02:22.294238 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:02:22.294211 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2\": container with ID starting with 49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2 not found: ID does not exist" containerID="49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2" Oct 31 01:02:22.294341 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.294248 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2"} err="failed to get container status \"49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2\": rpc error: code = NotFound desc = could not find container \"49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2\": container with ID starting with 49a6ccddfea070e2ca0a839f14f7b9cfef39ce02ce53b679212650db17f5f1d2 not found: ID does not exist" Oct 31 01:02:22.295608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.295583 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs\") pod \"b3cda834-3d91-4ae7-a139-daaadce7d091\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " Oct 31 01:02:22.295730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.295681 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache\") pod \"b3cda834-3d91-4ae7-a139-daaadce7d091\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " Oct 31 01:02:22.295815 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.295736 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home\") pod \"b3cda834-3d91-4ae7-a139-daaadce7d091\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " Oct 31 01:02:22.295877 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.295822 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4\" (UniqueName: \"kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4\") pod \"b3cda834-3d91-4ae7-a139-daaadce7d091\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " Oct 31 01:02:22.295933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.295890 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir\") pod \"b3cda834-3d91-4ae7-a139-daaadce7d091\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " Oct 31 01:02:22.296015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.296001 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm\") pod \"b3cda834-3d91-4ae7-a139-daaadce7d091\" (UID: \"b3cda834-3d91-4ae7-a139-daaadce7d091\") " Oct 31 01:02:22.296076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.295989 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache" pod "b3cda834-3d91-4ae7-a139-daaadce7d091" (UID: "b3cda834-3d91-4ae7-a139-daaadce7d091"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:22.296270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.296239 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir" pod "b3cda834-3d91-4ae7-a139-daaadce7d091" (UID: "b3cda834-3d91-4ae7-a139-daaadce7d091"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:22.296566 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.296545 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:22.296651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.296570 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:22.296651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.296631 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home" pod "b3cda834-3d91-4ae7-a139-daaadce7d091" (UID: "b3cda834-3d91-4ae7-a139-daaadce7d091"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:22.298111 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.298085 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs" pod "b3cda834-3d91-4ae7-a139-daaadce7d091" (UID: "b3cda834-3d91-4ae7-a139-daaadce7d091"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:22.298316 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.298290 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4" pod "b3cda834-3d91-4ae7-a139-daaadce7d091" (UID: "b3cda834-3d91-4ae7-a139-daaadce7d091"). InnerVolumeSpecName "kube-api-access-6scx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:22.298391 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.298366 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm" pod "b3cda834-3d91-4ae7-a139-daaadce7d091" (UID: "b3cda834-3d91-4ae7-a139-daaadce7d091"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:22.397827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.397783 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:22.397827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.397823 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cda834-3d91-4ae7-a139-daaadce7d091-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:22.397827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.397833 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cda834-3d91-4ae7-a139-daaadce7d091-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:22.398076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.397843 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6scx4\" (UniqueName: \"kubernetes.io/projected/b3cda834-3d91-4ae7-a139-daaadce7d091-kube-api-access-6scx4\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:22.512006 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.511972 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 01:02:22.512425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.512039 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 01:02:22.708419 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.708387 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w"] Oct 31 01:02:22.711949 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:22.711924 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w"] Oct 31 01:02:23.070080 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:23.070045 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:23.155205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:23.154278 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-16b984/auth-test-model-16b984-kserve-5554cddd5-wbz2w" podUID="b3cda834-3d91-4ae7-a139-daaadce7d091" containerName="main" probeResult="failure" output="Get \"https://10.134.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:02:23.511638 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:23.511610 5943 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 01:02:23.511808 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:23.511664 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 01:02:24.060302 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:24.060253 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cda834-3d91-4ae7-a139-daaadce7d091" path="/var/lib/kubelet/pods/b3cda834-3d91-4ae7-a139-daaadce7d091/volumes" Oct 31 01:02:24.515854 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:24.515825 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:02:24.516123 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:24.516104 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:02:24.516929 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:24.516910 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:02:27.477639 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:02:27.477599 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:02:27.840993 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:27.840957 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:31.794839 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:31.794799 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:02:32.560288 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:32.560250 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:35.949732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:35.949696 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp"] Oct 31 01:02:35.955807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:35.955778 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:36.648714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:36.648680 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.229026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.228991 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz"] Oct 31 01:02:37.229764 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.229743 5943 cpu_manager.go:414] "containerMap: removing container" podUID="17889338-f43a-4df9-b0f4-077d82386c39" containerName="revoke-keys" Oct 31 01:02:37.229881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.229767 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17889338-f43a-4df9-b0f4-077d82386c39" containerName="revoke-keys" Oct 31 01:02:37.229881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.229780 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b3cda834-3d91-4ae7-a139-daaadce7d091" containerName="main" Oct 31 01:02:37.229881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.229787 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cda834-3d91-4ae7-a139-daaadce7d091" containerName="main" Oct 31 01:02:37.229881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.229829 5943 cpu_manager.go:414] "containerMap: removing container" podUID="780306a8-3607-4c87-8343-f7824cca0b6a" containerName="main" Oct 31 01:02:37.229881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.229837 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780306a8-3607-4c87-8343-f7824cca0b6a" containerName="main" Oct 31 01:02:37.230139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.230003 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="780306a8-3607-4c87-8343-f7824cca0b6a" containerName="main" Oct 31 01:02:37.230139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.230028 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="17889338-f43a-4df9-b0f4-077d82386c39" containerName="revoke-keys" Oct 31 01:02:37.230139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.230038 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cda834-3d91-4ae7-a139-daaadce7d091" containerName="main" Oct 31 01:02:37.232869 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.232845 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.316489 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.316430 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5498e8-openshift-default-dockercfg-dlvq5\"" Oct 31 01:02:37.357860 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.357814 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz"] Oct 31 01:02:37.359802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.359654 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.388149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388116 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388328 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388266 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388403 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388328 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388475 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388459 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388597 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388672 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388647 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388735 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388716 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388787 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388750 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.388919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.388895 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvrr\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490315 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490214 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490315 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490347 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490384 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490414 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490502 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvrr\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490562 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490637 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490673 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.490976 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490782 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.491039 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.490991 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.491433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.491407 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.491544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.491499 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.491674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.491657 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.493278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.493225 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.495484 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.495458 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.513384 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.513349 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.551731 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.551700 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.551871 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.551772 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvrr\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr\") pod \"e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:37.759278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.759242 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.848361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:37.848326 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:38.286802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.286771 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz"] Oct 31 01:02:38.288499 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:02:38.288474 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e090d2_a606_4394_b9d4_a2f56bc255a8.slice/crio-15d8c741d11e1134ef90fc78af2f92a94204208db91d4350ab64e76cec0f1fbb WatchSource:0}: Error finding container 15d8c741d11e1134ef90fc78af2f92a94204208db91d4350ab64e76cec0f1fbb: Status 404 returned error can't find the container with id 15d8c741d11e1134ef90fc78af2f92a94204208db91d4350ab64e76cec0f1fbb Oct 31 01:02:38.292222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.292063 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:02:38.292222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.292145 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:02:38.292834 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.292401 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:02:38.433784 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.433744 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" event={"ID":"e4e090d2-a606-4394-b9d4-a2f56bc255a8","Type":"ContainerStarted","Data":"249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09"} Oct 31 01:02:38.433972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.433814 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" event={"ID":"e4e090d2-a606-4394-b9d4-a2f56bc255a8","Type":"ContainerStarted","Data":"15d8c741d11e1134ef90fc78af2f92a94204208db91d4350ab64e76cec0f1fbb"} Oct 31 01:02:38.554531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.554281 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" podStartSLOduration=1.554261351 podStartE2EDuration="1.554261351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:38.553246979 +0000 UTC m=+3538.646537206" watchObservedRunningTime="2026-10-31 01:02:38.554261351 +0000 UTC m=+3538.647551640" Oct 31 01:02:38.849071 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:38.848975 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:39.089875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:39.089840 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:39.777208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:39.776959 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:39.855319 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:39.855291 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:40.146705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:40.146605 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:40.453919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:40.453824 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:40.454750 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:40.454731 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:02:41.457757 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:41.457720 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:41.673202 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:41.669485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:42.339361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:42.339319 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.078682 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:44.078647 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.268042 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:44.268004 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.707680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:44.707649 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.863014 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:44.862974 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.028425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.028383 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.205748 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.205714 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.325057 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.324971 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt"] Oct 31 01:02:45.332163 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.332125 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.373056 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.373024 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt"] Oct 31 01:02:45.419494 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.419462 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5498e8-dockercfg-ms75q\"" Oct 31 01:02:45.419779 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.419521 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5498e8\"" Oct 31 01:02:45.420153 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.420136 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2msk\" (UniqueName: \"kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk\") pod \"payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.420364 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.420341 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.521433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.521397 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.521612 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.521563 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2msk\" (UniqueName: \"kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk\") pod \"payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.522125 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.522105 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.582733 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.582663 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2msk\" (UniqueName: \"kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk\") pod \"payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:45.652058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:45.652029 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:46.032954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:46.032926 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt"] Oct 31 01:02:46.034343 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:02:46.034302 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0766fd33_d97f_40ae_b0da_d51f3b64771d.slice/crio-ab78ab0a50f911c13d4e970cbfa0b80ac9fbbe7da8b43dc86c1b7189d9d6a386 WatchSource:0}: Error finding container ab78ab0a50f911c13d4e970cbfa0b80ac9fbbe7da8b43dc86c1b7189d9d6a386: Status 404 returned error can't find the container with id ab78ab0a50f911c13d4e970cbfa0b80ac9fbbe7da8b43dc86c1b7189d9d6a386 Oct 31 01:02:46.495758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:46.495667 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" event={"ID":"0766fd33-d97f-40ae-b0da-d51f3b64771d","Type":"ContainerStarted","Data":"2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb"} Oct 31 01:02:46.495758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:46.495731 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" event={"ID":"0766fd33-d97f-40ae-b0da-d51f3b64771d","Type":"ContainerStarted","Data":"ab78ab0a50f911c13d4e970cbfa0b80ac9fbbe7da8b43dc86c1b7189d9d6a386"} Oct 31 01:02:46.496249 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:46.495861 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:46.613616 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:46.613557 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" podStartSLOduration=1.613544247 podStartE2EDuration="1.613544247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:46.610360954 +0000 UTC m=+3546.703651178" watchObservedRunningTime="2026-10-31 01:02:46.613544247 +0000 UTC m=+3546.706834463" Oct 31 01:02:47.486001 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:47.485963 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:48.838297 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:48.838261 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:50.971697 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:50.971651 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" podUID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" containerName="main" containerID="cri-o://3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e" gracePeriod=45 Oct 31 01:02:51.240557 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.240484 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 01:02:51.296073 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296028 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs\") pod \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " Oct 31 01:02:51.296299 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296113 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home\") pod \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " Oct 31 01:02:51.296299 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296161 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir\") pod \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " Oct 31 01:02:51.296438 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296301 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm\") pod \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " Oct 31 01:02:51.296438 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296351 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd\" (UniqueName: \"kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd\") pod \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " Oct 31 01:02:51.296438 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296426 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home" pod "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" (UID: "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.296606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296444 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache\") pod \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\" (UID: \"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e\") " Oct 31 01:02:51.296606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296468 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir" pod "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" (UID: "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.296726 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296696 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache" pod "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" (UID: "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.296972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296953 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:51.296972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296974 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:51.297132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.296991 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:51.298651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.298628 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs" pod "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" (UID: "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:51.298754 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.298644 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm" pod "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" (UID: "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:51.298754 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.298666 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd" pod "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" (UID: "ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e"). InnerVolumeSpecName "kube-api-access-m8ccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:51.398395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.398346 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:51.398395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.398391 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ccd\" (UniqueName: \"kubernetes.io/projected/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-kube-api-access-m8ccd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:51.398395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.398406 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:51.529101 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.529059 5943 generic.go:356] "Generic (PLEG): container finished" podID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" containerID="3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e" exitCode=0 Oct 31 01:02:51.529318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.529109 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" Oct 31 01:02:51.529318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.529128 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" event={"ID":"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e","Type":"ContainerDied","Data":"3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e"} Oct 31 01:02:51.529318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.529166 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" event={"ID":"ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e","Type":"ContainerDied","Data":"86a22d6cc634458a2459694bbeb970d20a3c11e2fb52289079e3d7a920bf6c34"} Oct 31 01:02:51.529318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.529213 5943 scope.go:122] "RemoveContainer" containerID="3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e" Oct 31 01:02:51.539850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.539833 5943 scope.go:122] "RemoveContainer" containerID="3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e" Oct 31 01:02:51.540140 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:02:51.540121 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e\": container with ID starting with 3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e not found: ID does not exist" containerID="3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e" Oct 31 01:02:51.540218 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.540148 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e"} err="failed to get container status \"3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e\": rpc error: code = NotFound desc = could not find container \"3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e\": container with ID starting with 3145e9ec57bbf88219d51c3d83459a4b2bc4e44cf8059ee1aa05afa2d198211e not found: ID does not exist" Oct 31 01:02:51.693763 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.693731 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp"] Oct 31 01:02:51.699092 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:51.699063 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp"] Oct 31 01:02:52.042010 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.041969 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" path="/var/lib/kubelet/pods/ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e/volumes" Oct 31 01:02:52.059785 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.059755 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:52.202240 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.202191 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-093220/auth-test-model-093220-kserve-9c4c8ccdf-n2dtp" podUID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:02:52.236824 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.236786 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:52.472498 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.472410 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh"] Oct 31 01:02:52.473085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.473062 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" containerName="main" Oct 31 01:02:52.473246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.473087 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" containerName="main" Oct 31 01:02:52.473318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.473279 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4604a1-40d2-4f6d-a326-d3bd1a45fe4e" containerName="main" Oct 31 01:02:52.478021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.477999 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.513807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.513781 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktlk\" (UniqueName: \"kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk\") pod \"payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.514036 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.514017 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.533076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.533042 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2aebbd\"" Oct 31 01:02:52.534238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.534216 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2aebbd-dockercfg-hpzn4\"" Oct 31 01:02:52.535915 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.535890 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh"] Oct 31 01:02:52.617205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.616281 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.617205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.616351 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktlk\" (UniqueName: \"kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk\") pod \"payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.621554 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.617738 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.622585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.622320 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt"] Oct 31 01:02:52.634144 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.632291 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.674638 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.674608 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:52.677279 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.677248 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktlk\" (UniqueName: \"kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk\") pod \"payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.684981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.684954 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt"] Oct 31 01:02:52.717903 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.717860 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.718131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.718115 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxlf\" (UniqueName: \"kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf\") pod \"payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.792019 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.791990 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:52.819653 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.819612 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxlf\" (UniqueName: \"kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf\") pod \"payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.819809 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.819722 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.820499 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.820470 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.880829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.880795 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxlf\" (UniqueName: \"kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf\") pod \"payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:52.955070 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:52.955039 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:53.169375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.169346 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh"] Oct 31 01:02:53.170345 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:02:53.170311 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ae031a_b0ea_4426_873b_98e350e1d894.slice/crio-e97f49fea6561341c98159f70a7cb39a14689b577b7094ad5a3190b5e4cee495 WatchSource:0}: Error finding container e97f49fea6561341c98159f70a7cb39a14689b577b7094ad5a3190b5e4cee495: Status 404 returned error can't find the container with id e97f49fea6561341c98159f70a7cb39a14689b577b7094ad5a3190b5e4cee495 Oct 31 01:02:53.333976 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.333940 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt"] Oct 31 01:02:53.338580 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:02:53.338554 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1078191c_2ec0_4856_8861_8a19e35f03d5.slice/crio-1ff261fd686037805a1f33b17f0ef7554884bb2a30fba1d7442a34a39cc36c80 WatchSource:0}: Error finding container 1ff261fd686037805a1f33b17f0ef7554884bb2a30fba1d7442a34a39cc36c80: Status 404 returned error can't find the container with id 1ff261fd686037805a1f33b17f0ef7554884bb2a30fba1d7442a34a39cc36c80 Oct 31 01:02:53.548449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.548409 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" event={"ID":"1078191c-2ec0-4856-8861-8a19e35f03d5","Type":"ContainerStarted","Data":"f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d"} Oct 31 01:02:53.548665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.548471 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" event={"ID":"1078191c-2ec0-4856-8861-8a19e35f03d5","Type":"ContainerStarted","Data":"1ff261fd686037805a1f33b17f0ef7554884bb2a30fba1d7442a34a39cc36c80"} Oct 31 01:02:53.548665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.548498 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:02:53.550128 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.550087 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" event={"ID":"05ae031a-b0ea-4426-873b-98e350e1d894","Type":"ContainerStarted","Data":"368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f"} Oct 31 01:02:53.550266 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.550155 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:02:53.550266 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.550207 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" event={"ID":"05ae031a-b0ea-4426-873b-98e350e1d894","Type":"ContainerStarted","Data":"e97f49fea6561341c98159f70a7cb39a14689b577b7094ad5a3190b5e4cee495"} Oct 31 01:02:53.665149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.665099 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" podStartSLOduration=1.665085152 podStartE2EDuration="1.665085152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:53.662944727 +0000 UTC m=+3553.756234947" watchObservedRunningTime="2026-10-31 01:02:53.665085152 +0000 UTC m=+3553.758375340" Oct 31 01:02:53.781061 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:53.780997 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" podStartSLOduration=1.780979487 podStartE2EDuration="1.780979487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:53.779922376 +0000 UTC m=+3553.873212584" watchObservedRunningTime="2026-10-31 01:02:53.780979487 +0000 UTC m=+3553.874269674" Oct 31 01:02:54.026967 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:54.026939 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:57.475905 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:02:57.475865 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:02:57.504786 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:57.504760 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:02:57.805238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:57.805203 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:58.870688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:02:58.870645 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:02.647632 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:02.647595 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:04.560801 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:04.560770 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:03:04.561310 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:04.560846 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:03:07.384313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:07.384269 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:12.239659 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:12.239625 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:12.524311 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:12.524278 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:03:15.107743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:15.107700 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.301497 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:17.301456 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.506710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:17.506669 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.652235 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:17.652086 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.800988 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:17.800950 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:18.185991 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.185947 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg"] Oct 31 01:03:18.187100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.186855 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" containerID="cri-o://d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf" gracePeriod=30 Oct 31 01:03:18.347131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.347099 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr"] Oct 31 01:03:18.351797 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.351775 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.463006 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.459447 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-fae60c-openshift-default-dockercfg-t548m\"" Oct 31 01:03:18.464412 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.464384 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr"] Oct 31 01:03:18.469536 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.465635 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:18.474752 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.474598 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.475570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.475234 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.475955 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.475933 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.476433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.476412 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.476585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.476569 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.476733 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.476717 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.476854 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.476838 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.476980 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.476964 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.477122 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.477106 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhvz\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.578802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.578758 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.578845 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.578947 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579016 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579053 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579109 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579137 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579207 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579280 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhvz\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579334 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.579729 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.579705 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.580041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.580018 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.580149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.580057 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.580238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.580141 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.581809 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.581788 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.582357 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.582337 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.632685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.632649 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:18.640851 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.640820 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhvz\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.641015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.640919 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token\") pod \"e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:18.671761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:18.671728 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:19.101466 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.101422 5943 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:03:19.101669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.101488 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:03:19.116310 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.116285 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr"] Oct 31 01:03:19.117972 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:03:19.117938 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1875df41_6231_4298_9709_480d1fce66fa.slice/crio-397091fff28a27cd4bd63ad795e8ffd27e657bc9e9b0bfdae7c693b55cfb3d40 WatchSource:0}: Error finding container 397091fff28a27cd4bd63ad795e8ffd27e657bc9e9b0bfdae7c693b55cfb3d40: Status 404 returned error can't find the container with id 397091fff28a27cd4bd63ad795e8ffd27e657bc9e9b0bfdae7c693b55cfb3d40 Oct 31 01:03:19.120815 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.120776 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:19.120943 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.120846 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:19.120943 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.120879 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:19.730006 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.729958 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" event={"ID":"1875df41-6231-4298-9709-480d1fce66fa","Type":"ContainerStarted","Data":"a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879"} Oct 31 01:03:19.730495 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.730033 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" event={"ID":"1875df41-6231-4298-9709-480d1fce66fa","Type":"ContainerStarted","Data":"397091fff28a27cd4bd63ad795e8ffd27e657bc9e9b0bfdae7c693b55cfb3d40"} Oct 31 01:03:19.851493 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:19.851434 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podStartSLOduration=1.8514183960000001 podStartE2EDuration="1.851418396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:19.848651703 +0000 UTC m=+3579.941941940" watchObservedRunningTime="2026-10-31 01:03:19.851418396 +0000 UTC m=+3579.944708612" Oct 31 01:03:20.672901 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:20.672866 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:20.679244 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:20.679215 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:20.736860 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:20.736827 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:20.737814 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:20.737797 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:03:22.463809 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:22.463770 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:23.420907 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.420875 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:23.468041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.467976 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 01:03:23.548430 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548400 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548453 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548527 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548560 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548583 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548823 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548631 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548823 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548711 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548937 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548839 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.548937 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548883 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token\") pod \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\" (UID: \"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d\") " Oct 31 01:03:23.549048 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548834 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:23.549048 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548979 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:23.549048 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.548986 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:23.549241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549125 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:23.549314 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549249 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:23.549565 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549540 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.549850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549569 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.549850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549584 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.549850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549593 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.549850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.549602 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.551802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.551438 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "kube-api-access-mh5j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:23.551802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.551696 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:23.551802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.551761 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:23.552584 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.552564 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo" pod "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" (UID: "ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:03:23.600481 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.600448 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:23.651228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.651200 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.651228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.651225 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.651440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.651236 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh5j4\" (UniqueName: \"kubernetes.io/projected/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-kube-api-access-mh5j4\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.651440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.651247 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:23.753764 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.753666 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:23.763782 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.763742 5943 generic.go:356] "Generic (PLEG): container finished" podID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerID="d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf" exitCode=0 Oct 31 01:03:23.763987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.763793 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" Oct 31 01:03:23.763987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.763821 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" event={"ID":"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d","Type":"ContainerDied","Data":"d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf"} Oct 31 01:03:23.763987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.763890 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg" event={"ID":"ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d","Type":"ContainerDied","Data":"d6c5258f3dc6350b1acf7a414a2892668c44910db6a0e2d5805d7d650a1a1c90"} Oct 31 01:03:23.763987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.763925 5943 scope.go:122] "RemoveContainer" containerID="d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf" Oct 31 01:03:23.775648 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.775630 5943 scope.go:122] "RemoveContainer" containerID="d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf" Oct 31 01:03:23.775969 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:03:23.775950 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf\": container with ID starting with d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf not found: ID does not exist" containerID="d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf" Oct 31 01:03:23.776017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.775980 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf"} err="failed to get container status \"d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf\": rpc error: code = NotFound desc = could not find container \"d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf\": container with ID starting with d61795adebadd872680884736bd9429e0937451e1d057ac9e9afc70eecf196cf not found: ID does not exist" Oct 31 01:03:23.939332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.939288 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg"] Oct 31 01:03:23.948594 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:23.948564 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-rcbcg"] Oct 31 01:03:24.041387 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:24.041353 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" path="/var/lib/kubelet/pods/ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d/volumes" Oct 31 01:03:27.476964 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:03:27.476925 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:03:27.852388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:27.852354 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:28.200571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.200479 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:28.381928 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.381896 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:28.754203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.754028 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj"] Oct 31 01:03:28.762208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.754958 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" Oct 31 01:03:28.762208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.754981 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" Oct 31 01:03:28.762208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.755111 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7ee1fe-c3bb-49a3-b1be-d01b07dcee4d" containerName="istio-proxy" Oct 31 01:03:28.762208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.760934 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:28.814609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.814580 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fae60c\"" Oct 31 01:03:28.814609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.814582 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fae60c-dockercfg-mwlrl\"" Oct 31 01:03:28.837004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.836965 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj"] Oct 31 01:03:28.838232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.838206 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwhn\" (UniqueName: \"kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn\") pod \"payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:28.838381 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.838261 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:28.939493 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.939456 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwhn\" (UniqueName: \"kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn\") pod \"payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:28.939730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.939514 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:28.940284 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:28.940264 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:29.000594 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.000559 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwhn\" (UniqueName: \"kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn\") pod \"payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:29.099374 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.099339 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:29.474908 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.474878 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj"] Oct 31 01:03:29.476845 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:03:29.476818 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b1e695_efde_4463_80c7_0cd919b0829f.slice/crio-43fd0d15a7c979c055d4d21976c0953d0a051baca79297090ca90e035fa91d48 WatchSource:0}: Error finding container 43fd0d15a7c979c055d4d21976c0953d0a051baca79297090ca90e035fa91d48: Status 404 returned error can't find the container with id 43fd0d15a7c979c055d4d21976c0953d0a051baca79297090ca90e035fa91d48 Oct 31 01:03:29.818520 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.818484 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" event={"ID":"83b1e695-efde-4463-80c7-0cd919b0829f","Type":"ContainerStarted","Data":"62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32"} Oct 31 01:03:29.818959 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.818543 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" event={"ID":"83b1e695-efde-4463-80c7-0cd919b0829f","Type":"ContainerStarted","Data":"43fd0d15a7c979c055d4d21976c0953d0a051baca79297090ca90e035fa91d48"} Oct 31 01:03:29.818959 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.818666 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:29.934650 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:29.934454 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" podStartSLOduration=1.934436822 podStartE2EDuration="1.934436822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:29.933678689 +0000 UTC m=+3590.026968958" watchObservedRunningTime="2026-10-31 01:03:29.934436822 +0000 UTC m=+3590.027727043" Oct 31 01:03:33.029807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:33.029768 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:33.850058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:33.850024 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.228095 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:35.228061 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.872437 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:35.872398 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l"] Oct 31 01:03:35.878227 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:35.878197 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:35.931477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:35.931446 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-395b06-openshift-default-dockercfg-5zlxz\"" Oct 31 01:03:35.967239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:35.965325 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.984852 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:35.984802 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l"] Oct 31 01:03:36.039242 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.039169 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071006 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.070967 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071256 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.071053 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071256 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.071083 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071256 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.071242 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhbg\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.071511 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071603 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.071558 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.071703 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.071689 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.072053 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.072034 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.167651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.167553 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:36.174016 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.173974 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174024 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174055 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174098 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174121 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174165 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhbg\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174296 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174332 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.174525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.174374 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.175278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.175252 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.175691 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.175634 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.175691 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.175656 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.175850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.175824 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.175954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.175940 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.179209 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.179167 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.180132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.180110 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.258400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.258363 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.259914 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.259883 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhbg\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg\") pod \"e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.492879 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.492783 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:36.960982 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.960541 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l"] Oct 31 01:03:36.965073 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:03:36.965038 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b824e8_840c_4564_980a_80a4fdb5c205.slice/crio-2b7df1947e78178e5ed66259e78206ab2ce6c31b7263396159e7b80322d6a35c WatchSource:0}: Error finding container 2b7df1947e78178e5ed66259e78206ab2ce6c31b7263396159e7b80322d6a35c: Status 404 returned error can't find the container with id 2b7df1947e78178e5ed66259e78206ab2ce6c31b7263396159e7b80322d6a35c Oct 31 01:03:36.968001 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.967965 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:36.968113 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.968036 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:36.968113 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:36.968065 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:37.885635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:37.885603 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" event={"ID":"16b824e8-840c-4564-980a-80a4fdb5c205","Type":"ContainerStarted","Data":"be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6"} Oct 31 01:03:37.886097 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:37.885652 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" event={"ID":"16b824e8-840c-4564-980a-80a4fdb5c205","Type":"ContainerStarted","Data":"2b7df1947e78178e5ed66259e78206ab2ce6c31b7263396159e7b80322d6a35c"} Oct 31 01:03:38.094477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:38.094417 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podStartSLOduration=3.094403339 podStartE2EDuration="3.094403339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:38.09350035 +0000 UTC m=+3598.186790534" watchObservedRunningTime="2026-10-31 01:03:38.094403339 +0000 UTC m=+3598.187693605" Oct 31 01:03:38.492443 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:38.492408 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:38.493685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:38.493665 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:38.499780 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:38.499757 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:38.897158 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:38.897112 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:38.898979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:38.898949 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:03:40.849205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:40.836385 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:03:40.954807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:40.954763 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:41.117652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:41.117560 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:41.280543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:41.280505 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.653539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:43.653503 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:44.136150 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.136112 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:44.311251 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.311165 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:44.941746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.941713 5943 generic.go:356] "Generic (PLEG): container finished" podID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerID="e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d" exitCode=0 Oct 31 01:03:44.942246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.941800 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" event={"ID":"70fd23e6-8479-4811-aa12-3eb2a8521a28","Type":"ContainerDied","Data":"e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d"} Oct 31 01:03:44.942365 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.942348 5943 scope.go:122] "RemoveContainer" containerID="e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d" Oct 31 01:03:44.944449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.944410 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:44.944571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.944498 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:44.944571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:44.944551 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:03:45.969479 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:45.969444 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" event={"ID":"70fd23e6-8479-4811-aa12-3eb2a8521a28","Type":"ContainerStarted","Data":"cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06"} Oct 31 01:03:46.234283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:46.232536 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:03:46.254353 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:46.254317 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 01:03:46.254575 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:46.254379 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 01:03:47.235210 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:47.233526 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 01:03:47.235210 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:47.233594 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 01:03:48.251203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:48.250592 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 01:03:48.251203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:48.250683 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 01:03:48.615432 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:48.615386 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:49.253342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:49.253297 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 01:03:49.253905 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:49.253364 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 01:03:50.253208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:50.247609 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 01:03:50.253208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:50.247679 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 01:03:50.985098 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:50.985063 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:03:51.272478 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:51.272448 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:03:51.274143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:51.274125 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:03:53.421990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:53.421944 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:57.477011 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:03:57.476967 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:03:58.173763 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:58.173726 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:58.988944 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:03:58.988909 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:02.956831 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:02.956793 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:05.879635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:05.879600 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:04:07.691458 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:07.691421 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:12.338099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:12.338065 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:12.512877 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:12.512827 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:12.679304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:12.679218 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:17.346289 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:17.346248 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:20.048586 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:20.048555 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc"] Oct 31 01:04:20.097954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:20.097918 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100e23e9-sv2qc"] Oct 31 01:04:22.044066 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:22.044032 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01919f1b-4ca7-4458-b8bd-42eda44f3631" path="/var/lib/kubelet/pods/01919f1b-4ca7-4458-b8bd-42eda44f3631/volumes" Oct 31 01:04:22.098823 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:22.098794 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:22.248653 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:22.248609 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:22.393611 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:22.393509 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:27.030468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:27.030422 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:27.477391 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:04:27.477298 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:04:31.049251 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:31.049211 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv"] Oct 31 01:04:31.100865 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:31.100825 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b28b9d-bz2hv"] Oct 31 01:04:31.765246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:31.765211 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:32.042586 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:32.042500 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301b15f0-4336-4cf2-8c7e-b2cffcbb0a72" path="/var/lib/kubelet/pods/301b15f0-4336-4cf2-8c7e-b2cffcbb0a72/volumes" Oct 31 01:04:36.542845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:36.542779 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:41.325352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:41.325309 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:45.995540 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:45.995493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:50.764632 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:50.764593 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:51.373902 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:51.373857 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:51.516084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:51.516045 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.657634 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.657600 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw"] Oct 31 01:04:52.661687 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.661669 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.716074 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.716039 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:04:52.716305 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.716129 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:52.716384 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.716345 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:04:52.765582 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.765543 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:04:52.768477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.768448 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw"] Oct 31 01:04:52.808739 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.808703 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jhg\" (UniqueName: \"kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.808975 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.808772 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.809059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.808963 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.910865 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.910773 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.911026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.910903 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jhg\" (UniqueName: \"kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.911026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.910947 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.911647 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.911630 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.913323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.913298 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.971961 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.971932 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jhg\" (UniqueName: \"kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg\") pod \"maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:52.975624 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:52.975603 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:53.354427 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:53.354395 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw"] Oct 31 01:04:53.355968 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:04:53.355940 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb70792f_2456_40e7_b5ea_4d6b0c31cbcf.slice/crio-160e73b39625af98d5c4e84907a184467fe0914175bf434a175c7f78c842f20e WatchSource:0}: Error finding container 160e73b39625af98d5c4e84907a184467fe0914175bf434a175c7f78c842f20e: Status 404 returned error can't find the container with id 160e73b39625af98d5c4e84907a184467fe0914175bf434a175c7f78c842f20e Oct 31 01:04:53.445477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:53.445433 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" event={"ID":"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf","Type":"ContainerStarted","Data":"160e73b39625af98d5c4e84907a184467fe0914175bf434a175c7f78c842f20e"} Oct 31 01:04:54.462077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:54.461888 5943 generic.go:356] "Generic (PLEG): container finished" podID="eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" containerID="93b61cecc1a07abeea4c6437a544e110ee5836e0f05e786cbc51bf8d77d09a9e" exitCode=0 Oct 31 01:04:54.462077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:54.461965 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" event={"ID":"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf","Type":"ContainerDied","Data":"93b61cecc1a07abeea4c6437a544e110ee5836e0f05e786cbc51bf8d77d09a9e"} Oct 31 01:04:55.622794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.622769 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:55.714964 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.714932 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:55.745628 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.745544 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca\") pod \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " Oct 31 01:04:55.745788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.745654 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca\") pod \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " Oct 31 01:04:55.745847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.745780 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg\" (UniqueName: \"kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg\") pod \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\" (UID: \"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf\") " Oct 31 01:04:55.746079 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.746056 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca" pod "eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" (UID: "eb70792f-2456-40e7-b5ea-4d6b0c31cbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:55.746375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.746351 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:04:55.748236 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.748172 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca" pod "eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" (UID: "eb70792f-2456-40e7-b5ea-4d6b0c31cbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:55.748376 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.748324 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg" pod "eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" (UID: "eb70792f-2456-40e7-b5ea-4d6b0c31cbcf"). InnerVolumeSpecName "kube-api-access-k4jhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:55.847453 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.847416 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jhg\" (UniqueName: \"kubernetes.io/projected/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-kube-api-access-k4jhg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:04:55.847453 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.847445 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:04:55.947813 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:55.947772 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:56.101837 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:56.101793 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:56.476662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:56.476574 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" Oct 31 01:04:56.476662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:56.476611 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw" event={"ID":"eb70792f-2456-40e7-b5ea-4d6b0c31cbcf","Type":"ContainerDied","Data":"160e73b39625af98d5c4e84907a184467fe0914175bf434a175c7f78c842f20e"} Oct 31 01:04:56.476662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:56.476661 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="160e73b39625af98d5c4e84907a184467fe0914175bf434a175c7f78c842f20e" Oct 31 01:04:57.477687 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:04:57.477632 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:04:59.342318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:04:59.342279 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.367935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:00.367902 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.531765 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:00.531713 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.610813 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:00.610782 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:05:00.806596 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:00.806563 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:03.822954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:03.822916 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:03.970813 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:03.970773 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:05.316877 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:05.316843 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:10.088096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:10.088062 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:14.820708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:14.820672 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:19.659855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:19.659766 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:24.418913 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:24.418873 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:24.620469 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:24.620429 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:27.483917 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:05:27.483878 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:05:29.465186 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:29.465149 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:32.731403 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:32.731313 5943 generic.go:356] "Generic (PLEG): container finished" podID="1875df41-6231-4298-9709-480d1fce66fa" containerID="a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879" exitCode=0 Oct 31 01:05:32.731942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:32.731393 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" event={"ID":"1875df41-6231-4298-9709-480d1fce66fa","Type":"ContainerDied","Data":"a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879"} Oct 31 01:05:32.731942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:32.731922 5943 scope.go:122] "RemoveContainer" containerID="a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879" Oct 31 01:05:32.733580 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:32.733551 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:05:32.733699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:32.733635 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:05:32.733699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:32.733682 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:05:33.748277 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:33.748222 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" event={"ID":"1875df41-6231-4298-9709-480d1fce66fa","Type":"ContainerStarted","Data":"0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6"} Oct 31 01:05:34.356720 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:34.356678 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:34.672887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:34.672798 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:05:34.673918 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:34.673889 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 01:05:34.674045 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:34.673960 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 01:05:35.672557 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:35.672515 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 01:05:35.673063 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:35.672580 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 01:05:35.753210 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:35.747275 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:05:36.673041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:36.673007 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 01:05:36.673511 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:36.673063 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 01:05:37.672859 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:37.672820 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 01:05:37.673050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:37.672879 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 01:05:38.672768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:38.672733 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 01:05:38.673264 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:38.672791 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 01:05:39.207945 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:39.207906 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:39.677359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:39.677328 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:05:39.678707 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:39.678686 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:05:41.475014 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.474975 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f"] Oct 31 01:05:41.475710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.475694 5943 cpu_manager.go:414] "containerMap: removing container" podUID="eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" containerName="revoke-keys" Oct 31 01:05:41.475806 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.475711 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" containerName="revoke-keys" Oct 31 01:05:41.475845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.475828 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" containerName="revoke-keys" Oct 31 01:05:41.479875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.479853 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.534028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.533977 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:41.534028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.533989 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:05:41.534028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.533993 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:41.534398 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.534143 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:41.537748 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.537715 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f"] Oct 31 01:05:41.554766 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.554733 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7t2\" (UniqueName: \"kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.555008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.554986 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.555093 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.555047 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.656629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.656586 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.656629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.656635 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.656943 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.656698 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7t2\" (UniqueName: \"kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.657338 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.657311 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.659440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.659418 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.718115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.718083 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7t2\" (UniqueName: \"kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2\") pod \"maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:41.771505 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.771475 5943 scope.go:122] "RemoveContainer" containerID="bf68d2f597d8935886e23dd6478db7477cf000bbf5c085f1e3642acaeb00a1c4" Oct 31 01:05:41.781683 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.781661 5943 scope.go:122] "RemoveContainer" containerID="4bbb7a4f5a2f5c29a7e4274069c915e0a962abbc287efa6cd5743ae23d1f0d9e" Oct 31 01:05:41.796381 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:41.796361 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:42.183241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:42.183212 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f"] Oct 31 01:05:42.184406 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:05:42.184376 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76353cda_5c59_4731_adbd_50d3924edd06.slice/crio-22ba21bffacfb5b5130e5513cd2239666c6c702eb0f1bd97b51ae54f03f8d902 WatchSource:0}: Error finding container 22ba21bffacfb5b5130e5513cd2239666c6c702eb0f1bd97b51ae54f03f8d902: Status 404 returned error can't find the container with id 22ba21bffacfb5b5130e5513cd2239666c6c702eb0f1bd97b51ae54f03f8d902 Oct 31 01:05:42.818856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:42.818814 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" event={"ID":"76353cda-5c59-4731-adbd-50d3924edd06","Type":"ContainerStarted","Data":"f4a69942693db4ecf5a1dc14477e3591da30b229bc3d3bfdfedea0eaa3b4ea15"} Oct 31 01:05:42.819302 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:42.818877 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" event={"ID":"76353cda-5c59-4731-adbd-50d3924edd06","Type":"ContainerStarted","Data":"22ba21bffacfb5b5130e5513cd2239666c6c702eb0f1bd97b51ae54f03f8d902"} Oct 31 01:05:42.937559 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:42.937505 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" podStartSLOduration=1.9374903959999998 podStartE2EDuration="1.937490396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:42.934252523 +0000 UTC m=+3723.027542711" watchObservedRunningTime="2026-10-31 01:05:42.937490396 +0000 UTC m=+3723.030780582" Oct 31 01:05:43.035313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.035281 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj"] Oct 31 01:05:43.039405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.039385 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.098107 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.098074 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj"] Oct 31 01:05:43.175924 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.175884 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98f9f\" (UniqueName: \"kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.176085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.175928 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.176085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.176073 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.277643 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.277603 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98f9f\" (UniqueName: \"kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.277860 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.277647 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.277860 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.277699 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.278533 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.278510 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.280395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.280377 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.338946 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.338916 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98f9f\" (UniqueName: \"kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f\") pod \"maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.354235 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.354144 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:43.732862 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.732832 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj"] Oct 31 01:05:43.734376 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:05:43.734340 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d6dd44_a773_43e3_a5ac_4e254c2babf4.slice/crio-e05e7f6ffa00d25c31dec7ba3e529f3fa6414a14801d8fbc131f4c7de289ce8d WatchSource:0}: Error finding container e05e7f6ffa00d25c31dec7ba3e529f3fa6414a14801d8fbc131f4c7de289ce8d: Status 404 returned error can't find the container with id e05e7f6ffa00d25c31dec7ba3e529f3fa6414a14801d8fbc131f4c7de289ce8d Oct 31 01:05:43.828089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.828049 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" event={"ID":"02d6dd44-a773-43e3-a5ac-4e254c2babf4","Type":"ContainerStarted","Data":"e05e7f6ffa00d25c31dec7ba3e529f3fa6414a14801d8fbc131f4c7de289ce8d"} Oct 31 01:05:43.830003 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.829974 5943 generic.go:356] "Generic (PLEG): container finished" podID="76353cda-5c59-4731-adbd-50d3924edd06" containerID="f4a69942693db4ecf5a1dc14477e3591da30b229bc3d3bfdfedea0eaa3b4ea15" exitCode=0 Oct 31 01:05:43.830133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:43.830067 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" event={"ID":"76353cda-5c59-4731-adbd-50d3924edd06","Type":"ContainerDied","Data":"f4a69942693db4ecf5a1dc14477e3591da30b229bc3d3bfdfedea0eaa3b4ea15"} Oct 31 01:05:44.260010 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:44.259971 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:44.839330 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:44.839295 5943 generic.go:356] "Generic (PLEG): container finished" podID="02d6dd44-a773-43e3-a5ac-4e254c2babf4" containerID="b3ec7b9734f0fe6e1161325f464849f10255a704d1e144c24c896c9c5064fda4" exitCode=0 Oct 31 01:05:44.839739 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:44.839374 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" event={"ID":"02d6dd44-a773-43e3-a5ac-4e254c2babf4","Type":"ContainerDied","Data":"b3ec7b9734f0fe6e1161325f464849f10255a704d1e144c24c896c9c5064fda4"} Oct 31 01:05:44.947766 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:44.947732 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:05:44.998284 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:44.998253 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:45.100448 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.100364 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca\") pod \"76353cda-5c59-4731-adbd-50d3924edd06\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " Oct 31 01:05:45.100606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.100481 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca\") pod \"76353cda-5c59-4731-adbd-50d3924edd06\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " Oct 31 01:05:45.100606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.100549 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2\" (UniqueName: \"kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2\") pod \"76353cda-5c59-4731-adbd-50d3924edd06\" (UID: \"76353cda-5c59-4731-adbd-50d3924edd06\") " Oct 31 01:05:45.100906 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.100878 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca" pod "76353cda-5c59-4731-adbd-50d3924edd06" (UID: "76353cda-5c59-4731-adbd-50d3924edd06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:45.101168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.101146 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:45.102649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.102623 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca" pod "76353cda-5c59-4731-adbd-50d3924edd06" (UID: "76353cda-5c59-4731-adbd-50d3924edd06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:45.102765 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.102754 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2" pod "76353cda-5c59-4731-adbd-50d3924edd06" (UID: "76353cda-5c59-4731-adbd-50d3924edd06"). InnerVolumeSpecName "kube-api-access-fg7t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:45.203008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.202967 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353cda-5c59-4731-adbd-50d3924edd06-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:45.203008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.203004 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg7t2\" (UniqueName: \"kubernetes.io/projected/76353cda-5c59-4731-adbd-50d3924edd06-kube-api-access-fg7t2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:45.847889 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.847850 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" event={"ID":"76353cda-5c59-4731-adbd-50d3924edd06","Type":"ContainerDied","Data":"22ba21bffacfb5b5130e5513cd2239666c6c702eb0f1bd97b51ae54f03f8d902"} Oct 31 01:05:45.847889 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.847874 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f" Oct 31 01:05:45.848388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:45.847908 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ba21bffacfb5b5130e5513cd2239666c6c702eb0f1bd97b51ae54f03f8d902" Oct 31 01:05:46.005278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.005245 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:46.113816 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.113723 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca\") pod \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " Oct 31 01:05:46.113997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.113832 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f\" (UniqueName: \"kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f\") pod \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " Oct 31 01:05:46.113997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.113916 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca\") pod \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\" (UID: \"02d6dd44-a773-43e3-a5ac-4e254c2babf4\") " Oct 31 01:05:46.114380 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.114348 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca" pod "02d6dd44-a773-43e3-a5ac-4e254c2babf4" (UID: "02d6dd44-a773-43e3-a5ac-4e254c2babf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:46.114646 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.114611 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:46.116216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.116192 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca" pod "02d6dd44-a773-43e3-a5ac-4e254c2babf4" (UID: "02d6dd44-a773-43e3-a5ac-4e254c2babf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:46.116561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.116543 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f" pod "02d6dd44-a773-43e3-a5ac-4e254c2babf4" (UID: "02d6dd44-a773-43e3-a5ac-4e254c2babf4"). InnerVolumeSpecName "kube-api-access-98f9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:46.215635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.215593 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d6dd44-a773-43e3-a5ac-4e254c2babf4-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:46.215635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.215629 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98f9f\" (UniqueName: \"kubernetes.io/projected/02d6dd44-a773-43e3-a5ac-4e254c2babf4-kube-api-access-98f9f\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:46.860276 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.860242 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" Oct 31 01:05:46.860690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.860246 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj" event={"ID":"02d6dd44-a773-43e3-a5ac-4e254c2babf4","Type":"ContainerDied","Data":"e05e7f6ffa00d25c31dec7ba3e529f3fa6414a14801d8fbc131f4c7de289ce8d"} Oct 31 01:05:46.860690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:46.860372 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05e7f6ffa00d25c31dec7ba3e529f3fa6414a14801d8fbc131f4c7de289ce8d" Oct 31 01:05:49.169769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:49.169723 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:52.992471 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:52.992428 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt"] Oct 31 01:05:53.327382 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:53.327333 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:53.477416 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:53.477381 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:53.966275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:53.966231 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:54.940222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:54.939307 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:05:54.940222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:54.939395 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:05:55.458354 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:55.458313 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt"] Oct 31 01:05:55.564939 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:55.564908 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh"] Oct 31 01:05:55.816901 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:55.816866 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:56.006538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.006503 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:56.949534 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.949497 5943 generic.go:356] "Generic (PLEG): container finished" podID="16b824e8-840c-4564-980a-80a4fdb5c205" containerID="be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6" exitCode=0 Oct 31 01:05:56.949949 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.949925 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" event={"ID":"16b824e8-840c-4564-980a-80a4fdb5c205","Type":"ContainerDied","Data":"be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6"} Oct 31 01:05:56.950702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.950683 5943 scope.go:122] "RemoveContainer" containerID="be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6" Oct 31 01:05:56.965639 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.965603 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:05:56.965933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.965907 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:05:56.966125 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:56.966098 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:05:57.485860 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:05:57.485808 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:05:57.963585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:57.963549 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" event={"ID":"16b824e8-840c-4564-980a-80a4fdb5c205","Type":"ContainerStarted","Data":"e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843"} Oct 31 01:05:57.993172 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:57.993122 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" podUID="0766fd33-d97f-40ae-b0da-d51f3b64771d" containerName="payload-processing" containerID="cri-o://2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb" gracePeriod=25 Oct 31 01:05:58.302335 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.302129 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:58.506699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.505947 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:05:58.506699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.506266 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:05:58.506699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.506329 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:05:58.527543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.527304 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:05:58.741858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.741339 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume\") pod \"0766fd33-d97f-40ae-b0da-d51f3b64771d\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " Oct 31 01:05:58.742166 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.742143 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk\" (UniqueName: \"kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk\") pod \"0766fd33-d97f-40ae-b0da-d51f3b64771d\" (UID: \"0766fd33-d97f-40ae-b0da-d51f3b64771d\") " Oct 31 01:05:58.743157 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.741767 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume" pod "0766fd33-d97f-40ae-b0da-d51f3b64771d" (UID: "0766fd33-d97f-40ae-b0da-d51f3b64771d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:58.804002 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.803936 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk" pod "0766fd33-d97f-40ae-b0da-d51f3b64771d" (UID: "0766fd33-d97f-40ae-b0da-d51f3b64771d"). InnerVolumeSpecName "kube-api-access-d2msk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:58.871377 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.845214 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0766fd33-d97f-40ae-b0da-d51f3b64771d-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.871377 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.845260 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2msk\" (UniqueName: \"kubernetes.io/projected/0766fd33-d97f-40ae-b0da-d51f3b64771d-kube-api-access-d2msk\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.986328 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.986291 5943 generic.go:356] "Generic (PLEG): container finished" podID="0766fd33-d97f-40ae-b0da-d51f3b64771d" containerID="2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb" exitCode=0 Oct 31 01:05:58.986521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.986336 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" Oct 31 01:05:58.986521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.986348 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" event={"ID":"0766fd33-d97f-40ae-b0da-d51f3b64771d","Type":"ContainerDied","Data":"2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb"} Oct 31 01:05:58.986521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.986404 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt" event={"ID":"0766fd33-d97f-40ae-b0da-d51f3b64771d","Type":"ContainerDied","Data":"ab78ab0a50f911c13d4e970cbfa0b80ac9fbbe7da8b43dc86c1b7189d9d6a386"} Oct 31 01:05:58.986521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.986429 5943 scope.go:122] "RemoveContainer" containerID="2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb" Oct 31 01:05:58.997449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.997312 5943 scope.go:122] "RemoveContainer" containerID="2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb" Oct 31 01:05:58.997746 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:05:58.997718 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb\": container with ID starting with 2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb not found: ID does not exist" containerID="2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb" Oct 31 01:05:58.997872 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:58.997754 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb"} err="failed to get container status \"2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb\": rpc error: code = NotFound desc = could not find container \"2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb\": container with ID starting with 2aae8826c7cf272d3ec3658b64fdff7fd5d27bf9ac56e6cc1087cd9745f436eb not found: ID does not exist" Oct 31 01:05:59.171061 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:59.170239 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt"] Oct 31 01:05:59.181535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:59.181494 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5498e8-6984fc59c5-9tmmt"] Oct 31 01:05:59.495672 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:59.495640 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:05:59.495978 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:59.495940 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:05:59.776535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:59.776437 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:59.946725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:05:59.946690 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:00.082402 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.082305 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0766fd33-d97f-40ae-b0da-d51f3b64771d" path="/var/lib/kubelet/pods/0766fd33-d97f-40ae-b0da-d51f3b64771d/volumes" Oct 31 01:06:00.202566 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.202540 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:00.360398 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.360307 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:00.459103 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.459061 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" podUID="1078191c-2ec0-4856-8861-8a19e35f03d5" containerName="payload-processing" containerID="cri-o://f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d" gracePeriod=25 Oct 31 01:06:00.493573 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.493544 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:06:00.493678 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.493619 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:06:00.565481 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.565406 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" podUID="05ae031a-b0ea-4426-873b-98e350e1d894" containerName="payload-pre-processing" containerID="cri-o://368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f" gracePeriod=25 Oct 31 01:06:00.662032 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.662001 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f"] Oct 31 01:06:00.665766 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.665723 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5498e8-wqt6f"] Oct 31 01:06:00.806705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.806673 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:06:00.895600 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.895573 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:06:00.903806 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.903782 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume\") pod \"05ae031a-b0ea-4426-873b-98e350e1d894\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " Oct 31 01:06:00.903933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.903863 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf\" (UniqueName: \"kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf\") pod \"1078191c-2ec0-4856-8861-8a19e35f03d5\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " Oct 31 01:06:00.903933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.903907 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume\") pod \"1078191c-2ec0-4856-8861-8a19e35f03d5\" (UID: \"1078191c-2ec0-4856-8861-8a19e35f03d5\") " Oct 31 01:06:00.904005 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.903975 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk\" (UniqueName: \"kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk\") pod \"05ae031a-b0ea-4426-873b-98e350e1d894\" (UID: \"05ae031a-b0ea-4426-873b-98e350e1d894\") " Oct 31 01:06:00.904322 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.904279 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume" pod "05ae031a-b0ea-4426-873b-98e350e1d894" (UID: "05ae031a-b0ea-4426-873b-98e350e1d894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:00.904439 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.904341 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume" pod "1078191c-2ec0-4856-8861-8a19e35f03d5" (UID: "1078191c-2ec0-4856-8861-8a19e35f03d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:00.904601 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.904580 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ae031a-b0ea-4426-873b-98e350e1d894-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:00.904659 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.904606 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1078191c-2ec0-4856-8861-8a19e35f03d5-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:00.906364 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.906340 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk" pod "05ae031a-b0ea-4426-873b-98e350e1d894" (UID: "05ae031a-b0ea-4426-873b-98e350e1d894"). InnerVolumeSpecName "kube-api-access-pktlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:00.906653 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:00.906629 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf" pod "1078191c-2ec0-4856-8861-8a19e35f03d5" (UID: "1078191c-2ec0-4856-8861-8a19e35f03d5"). InnerVolumeSpecName "kube-api-access-ltxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:01.035264 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.013124 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktlk\" (UniqueName: \"kubernetes.io/projected/05ae031a-b0ea-4426-873b-98e350e1d894-kube-api-access-pktlk\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:01.035264 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.013162 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxlf\" (UniqueName: \"kubernetes.io/projected/1078191c-2ec0-4856-8861-8a19e35f03d5-kube-api-access-ltxlf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:01.102969 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.102931 5943 generic.go:356] "Generic (PLEG): container finished" podID="05ae031a-b0ea-4426-873b-98e350e1d894" containerID="368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f" exitCode=0 Oct 31 01:06:01.103286 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.103080 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" event={"ID":"05ae031a-b0ea-4426-873b-98e350e1d894","Type":"ContainerDied","Data":"368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f"} Oct 31 01:06:01.103401 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.103367 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" event={"ID":"05ae031a-b0ea-4426-873b-98e350e1d894","Type":"ContainerDied","Data":"e97f49fea6561341c98159f70a7cb39a14689b577b7094ad5a3190b5e4cee495"} Oct 31 01:06:01.103462 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.103423 5943 scope.go:122] "RemoveContainer" containerID="368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f" Oct 31 01:06:01.103538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.103134 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh" Oct 31 01:06:01.106350 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.106321 5943 generic.go:356] "Generic (PLEG): container finished" podID="1078191c-2ec0-4856-8861-8a19e35f03d5" containerID="f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d" exitCode=0 Oct 31 01:06:01.106440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.106394 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" event={"ID":"1078191c-2ec0-4856-8861-8a19e35f03d5","Type":"ContainerDied","Data":"f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d"} Oct 31 01:06:01.106440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.106435 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" event={"ID":"1078191c-2ec0-4856-8861-8a19e35f03d5","Type":"ContainerDied","Data":"1ff261fd686037805a1f33b17f0ef7554884bb2a30fba1d7442a34a39cc36c80"} Oct 31 01:06:01.106528 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.106515 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt" Oct 31 01:06:01.116794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.116754 5943 scope.go:122] "RemoveContainer" containerID="368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f" Oct 31 01:06:01.117313 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:01.117267 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f\": container with ID starting with 368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f not found: ID does not exist" containerID="368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f" Oct 31 01:06:01.117313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.117293 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f"} err="failed to get container status \"368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f\": rpc error: code = NotFound desc = could not find container \"368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f\": container with ID starting with 368b114f4bec2105041b4faa46fc58d761cd57b9d1f883b95735dbec1c74678f not found: ID does not exist" Oct 31 01:06:01.117313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.117309 5943 scope.go:122] "RemoveContainer" containerID="f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d" Oct 31 01:06:01.129910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.129894 5943 scope.go:122] "RemoveContainer" containerID="f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d" Oct 31 01:06:01.130729 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:01.130705 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d\": container with ID starting with f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d not found: ID does not exist" containerID="f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d" Oct 31 01:06:01.130883 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.130860 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d"} err="failed to get container status \"f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d\": rpc error: code = NotFound desc = could not find container \"f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d\": container with ID starting with f1ee910472aaa171c183ca101c8595809c3331cb311ff549cc9ae368d40ca77d not found: ID does not exist" Oct 31 01:06:01.156267 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.153151 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:01.271990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.271947 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh"] Oct 31 01:06:01.277231 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.277198 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2aebbd-58df5c5dff-42qgh"] Oct 31 01:06:01.386081 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.385989 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt"] Oct 31 01:06:01.400215 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.400157 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2aebbd-598945d587-gkztt"] Oct 31 01:06:01.493195 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.493149 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:06:01.493371 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:01.493227 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:06:02.057925 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:02.057879 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ae031a-b0ea-4426-873b-98e350e1d894" path="/var/lib/kubelet/pods/05ae031a-b0ea-4426-873b-98e350e1d894/volumes" Oct 31 01:06:02.058539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:02.058518 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1078191c-2ec0-4856-8861-8a19e35f03d5" path="/var/lib/kubelet/pods/1078191c-2ec0-4856-8861-8a19e35f03d5/volumes" Oct 31 01:06:02.059011 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:02.058991 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76353cda-5c59-4731-adbd-50d3924edd06" path="/var/lib/kubelet/pods/76353cda-5c59-4731-adbd-50d3924edd06/volumes" Oct 31 01:06:02.493357 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:02.493278 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:06:02.493357 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:02.493338 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:06:03.493856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:03.493811 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:06:03.494274 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:03.493886 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:06:04.500297 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:04.500264 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:06:04.502910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:04.500710 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:06:04.502910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:04.501627 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:06:04.994884 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:04.994847 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:05.139481 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:05.139449 5943 generic.go:356] "Generic (PLEG): container finished" podID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerID="249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09" exitCode=0 Oct 31 01:06:05.139709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:05.139530 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" event={"ID":"e4e090d2-a606-4394-b9d4-a2f56bc255a8","Type":"ContainerDied","Data":"249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09"} Oct 31 01:06:05.140293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:05.140269 5943 scope.go:122] "RemoveContainer" containerID="249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09" Oct 31 01:06:05.141996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:05.141965 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:06:05.142119 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:05.142041 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:06:05.142119 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:05.142080 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:06:06.163571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.163537 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" event={"ID":"e4e090d2-a606-4394-b9d4-a2f56bc255a8","Type":"ContainerStarted","Data":"69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674"} Oct 31 01:06:06.554890 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.554844 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:06.768516 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.768485 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:06.853414 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.853322 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:06:06.854708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.854680 5943 patch_prober.go:29] interesting pod/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 31 01:06:06.854852 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.854743 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 31 01:06:06.922701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:06.922662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:07.055449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:07.055413 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj"] Oct 31 01:06:07.059330 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:07.059305 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2aebbd-zdlwj"] Oct 31 01:06:07.501436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:07.501401 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:07.849650 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:07.849615 5943 patch_prober.go:29] interesting pod/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 31 01:06:07.849974 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:07.849943 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 31 01:06:07.949206 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:07.948304 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:08.059989 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:08.059957 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d6dd44-a773-43e3-a5ac-4e254c2babf4" path="/var/lib/kubelet/pods/02d6dd44-a773-43e3-a5ac-4e254c2babf4/volumes" Oct 31 01:06:08.156690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:08.156598 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:08.325708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:08.325674 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz"] Oct 31 01:06:08.325935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:08.325909 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" containerID="cri-o://69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674" gracePeriod=30 Oct 31 01:06:08.362200 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:08.362079 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:10.453828 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:10.453789 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:06:10.583817 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:10.583782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:11.152129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.152087 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h"] Oct 31 01:06:11.153351 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.153325 5943 cpu_manager.go:414] "containerMap: removing container" podUID="05ae031a-b0ea-4426-873b-98e350e1d894" containerName="payload-pre-processing" Oct 31 01:06:11.153529 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.153515 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ae031a-b0ea-4426-873b-98e350e1d894" containerName="payload-pre-processing" Oct 31 01:06:11.153641 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.153630 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1078191c-2ec0-4856-8861-8a19e35f03d5" containerName="payload-processing" Oct 31 01:06:11.153725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.153715 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1078191c-2ec0-4856-8861-8a19e35f03d5" containerName="payload-processing" Oct 31 01:06:11.153843 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.153832 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0766fd33-d97f-40ae-b0da-d51f3b64771d" containerName="payload-processing" Oct 31 01:06:11.153927 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.153918 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0766fd33-d97f-40ae-b0da-d51f3b64771d" containerName="payload-processing" Oct 31 01:06:11.154067 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154055 5943 cpu_manager.go:414] "containerMap: removing container" podUID="02d6dd44-a773-43e3-a5ac-4e254c2babf4" containerName="revoke-keys" Oct 31 01:06:11.154157 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154147 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d6dd44-a773-43e3-a5ac-4e254c2babf4" containerName="revoke-keys" Oct 31 01:06:11.154271 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154260 5943 cpu_manager.go:414] "containerMap: removing container" podUID="76353cda-5c59-4731-adbd-50d3924edd06" containerName="revoke-keys" Oct 31 01:06:11.154352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154342 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76353cda-5c59-4731-adbd-50d3924edd06" containerName="revoke-keys" Oct 31 01:06:11.154578 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154562 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="76353cda-5c59-4731-adbd-50d3924edd06" containerName="revoke-keys" Oct 31 01:06:11.154698 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154687 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ae031a-b0ea-4426-873b-98e350e1d894" containerName="payload-pre-processing" Oct 31 01:06:11.154784 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154775 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1078191c-2ec0-4856-8861-8a19e35f03d5" containerName="payload-processing" Oct 31 01:06:11.154868 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154859 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0766fd33-d97f-40ae-b0da-d51f3b64771d" containerName="payload-processing" Oct 31 01:06:11.154946 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.154937 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d6dd44-a773-43e3-a5ac-4e254c2babf4" containerName="revoke-keys" Oct 31 01:06:11.160650 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.160626 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.219328 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.218568 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3853b6-openshift-default-dockercfg-n255q\"" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.235996 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236072 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79g24\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236109 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236226 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236286 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236343 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236377 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236439 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.236770 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.236470 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.276629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.276420 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:11.284578 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.284548 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h"] Oct 31 01:06:11.338145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.338103 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.338538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.338510 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.338819 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.338798 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.338954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.338938 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.339099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.339085 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.339242 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.339227 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.339581 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.339540 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.339793 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.339724 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79g24\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.339934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.339916 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.340574 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.340552 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.340993 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.340971 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.341788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.341766 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.344062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.342998 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.344062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.343296 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.346681 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.346660 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.349208 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.349118 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.405531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.405451 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79g24\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.405531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.405515 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token\") pod \"e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.477811 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.477778 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:11.530021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.529979 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:11.931578 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.931550 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h"] Oct 31 01:06:11.933633 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:11.933604 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf5c579_b3d6_429b_b91b_b45c109dfb09.slice/crio-43365d08a1fb4e389b2d0af2ee4000902dbbe3180a2269da72af61c91ed45602 WatchSource:0}: Error finding container 43365d08a1fb4e389b2d0af2ee4000902dbbe3180a2269da72af61c91ed45602: Status 404 returned error can't find the container with id 43365d08a1fb4e389b2d0af2ee4000902dbbe3180a2269da72af61c91ed45602 Oct 31 01:06:11.936532 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.936495 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:06:11.936655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.936561 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:06:11.936655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:11.936594 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:06:12.008639 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:12.008592 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:12.231371 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:12.231279 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" event={"ID":"ecf5c579-b3d6-429b-b91b-b45c109dfb09","Type":"ContainerStarted","Data":"b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f"} Oct 31 01:06:12.231371 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:12.231341 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" event={"ID":"ecf5c579-b3d6-429b-b91b-b45c109dfb09","Type":"ContainerStarted","Data":"43365d08a1fb4e389b2d0af2ee4000902dbbe3180a2269da72af61c91ed45602"} Oct 31 01:06:12.356487 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:12.356427 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podStartSLOduration=1.356404207 podStartE2EDuration="1.356404207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:12.351544666 +0000 UTC m=+3752.444834899" watchObservedRunningTime="2026-10-31 01:06:12.356404207 +0000 UTC m=+3752.449694396" Oct 31 01:06:12.478284 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:12.478239 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:13.483953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:13.483920 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:13.579962 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:13.579923 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:13.751735 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:13.751636 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:13.957026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:13.956978 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:14.244160 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:14.244128 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:14.245086 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:14.245066 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:06:15.222251 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:15.222194 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:15.457674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:15.457637 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:16.491638 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.491590 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm"] Oct 31 01:06:16.495812 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.495793 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.554496 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.554458 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm"] Oct 31 01:06:16.599990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.599961 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:16.599990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.599955 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:16.600234 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.600047 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:16.600234 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.600061 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:06:16.612230 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.612204 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cwg\" (UniqueName: \"kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.612392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.612243 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.612392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.612343 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.713518 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.713487 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cwg\" (UniqueName: \"kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.713652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.713528 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.713652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.713572 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.714406 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.714343 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.715981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.715959 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.721205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.721186 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:06:16.774636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.774610 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cwg\" (UniqueName: \"kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.811115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.811086 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:16.814156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814135 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814174 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814241 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814381 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814518 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814422 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814518 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814453 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814639 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814626 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.814736 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814709 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:16.814986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814942 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:16.815086 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.814980 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:16.815145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.815007 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.815319 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.815278 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:16.815478 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.815408 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy\") pod \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\" (UID: \"e4e090d2-a606-4394-b9d4-a2f56bc255a8\") " Oct 31 01:06:16.816129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.815979 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.816129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.816054 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.816129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.816071 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.816129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.816086 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.816129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.816063 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:16.816891 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.816861 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "kube-api-access-vfvrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:16.817088 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.817070 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:16.817813 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.817795 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:16.817986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.817959 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo" pod "e4e090d2-a606-4394-b9d4-a2f56bc255a8" (UID: "e4e090d2-a606-4394-b9d4-a2f56bc255a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:16.917934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.917897 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.917934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.917938 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvrr\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-kube-api-access-vfvrr\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.918216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.917956 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.918216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.917970 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e090d2-a606-4394-b9d4-a2f56bc255a8-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:16.918216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:16.917983 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e090d2-a606-4394-b9d4-a2f56bc255a8-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:17.195450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.194409 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm"] Oct 31 01:06:17.195450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.195268 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:17.199857 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:17.199830 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6de869_5676_4e60_ac9b_dde7a51b7cd6.slice/crio-5fcf4a90e93d78807ac21156ba2669a99774daa697cf11da5a39d4e515a3408e WatchSource:0}: Error finding container 5fcf4a90e93d78807ac21156ba2669a99774daa697cf11da5a39d4e515a3408e: Status 404 returned error can't find the container with id 5fcf4a90e93d78807ac21156ba2669a99774daa697cf11da5a39d4e515a3408e Oct 31 01:06:17.266264 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.266214 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" event={"ID":"5c6de869-5676-4e60-ac9b-dde7a51b7cd6","Type":"ContainerStarted","Data":"5fcf4a90e93d78807ac21156ba2669a99774daa697cf11da5a39d4e515a3408e"} Oct 31 01:06:17.268144 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.268114 5943 generic.go:356] "Generic (PLEG): container finished" podID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerID="69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674" exitCode=0 Oct 31 01:06:17.268323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.268173 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" Oct 31 01:06:17.268323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.268224 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" event={"ID":"e4e090d2-a606-4394-b9d4-a2f56bc255a8","Type":"ContainerDied","Data":"69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674"} Oct 31 01:06:17.268323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.268284 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz" event={"ID":"e4e090d2-a606-4394-b9d4-a2f56bc255a8","Type":"ContainerDied","Data":"15d8c741d11e1134ef90fc78af2f92a94204208db91d4350ab64e76cec0f1fbb"} Oct 31 01:06:17.268538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.268314 5943 scope.go:122] "RemoveContainer" containerID="69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674" Oct 31 01:06:17.279095 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.279079 5943 scope.go:122] "RemoveContainer" containerID="249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09" Oct 31 01:06:17.288919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.288901 5943 scope.go:122] "RemoveContainer" containerID="69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674" Oct 31 01:06:17.289335 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:17.289271 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674\": container with ID starting with 69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674 not found: ID does not exist" containerID="69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674" Oct 31 01:06:17.289454 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.289345 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674"} err="failed to get container status \"69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674\": rpc error: code = NotFound desc = could not find container \"69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674\": container with ID starting with 69df31936839577392075d79aca1e831bc617df3e20b4412b7b6c3582df35674 not found: ID does not exist" Oct 31 01:06:17.289454 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.289384 5943 scope.go:122] "RemoveContainer" containerID="249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09" Oct 31 01:06:17.289780 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:17.289752 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09\": container with ID starting with 249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09 not found: ID does not exist" containerID="249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09" Oct 31 01:06:17.289867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.289797 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09"} err="failed to get container status \"249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09\": rpc error: code = NotFound desc = could not find container \"249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09\": container with ID starting with 249f8a83d0d844903b1415201ee41dd28b19a2ac059901d15685e8ea97220e09 not found: ID does not exist" Oct 31 01:06:17.451230 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.451135 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz"] Oct 31 01:06:17.460896 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:17.460868 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5498e8-openshift-default-55dbb745f7-brtxz"] Oct 31 01:06:18.049018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:18.048978 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" path="/var/lib/kubelet/pods/e4e090d2-a606-4394-b9d4-a2f56bc255a8/volumes" Oct 31 01:06:18.277370 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:18.277338 5943 generic.go:356] "Generic (PLEG): container finished" podID="5c6de869-5676-4e60-ac9b-dde7a51b7cd6" containerID="1af40e9fb3fdc71f7c45bb9a45db2f950b6cf5aaeb4994c307ef56d136d5b192" exitCode=0 Oct 31 01:06:18.277595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:18.277424 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" event={"ID":"5c6de869-5676-4e60-ac9b-dde7a51b7cd6","Type":"ContainerDied","Data":"1af40e9fb3fdc71f7c45bb9a45db2f950b6cf5aaeb4994c307ef56d136d5b192"} Oct 31 01:06:19.027591 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.027537 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:19.203096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.203052 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:19.437430 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.437401 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn"] Oct 31 01:06:19.438005 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.437989 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" Oct 31 01:06:19.438072 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.438006 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" Oct 31 01:06:19.438140 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.438131 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" Oct 31 01:06:19.441714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.441691 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.466465 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.466442 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:19.497274 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.497242 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3853b6-dockercfg-lh5rw\"" Oct 31 01:06:19.582399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.558322 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3853b6\"" Oct 31 01:06:19.582399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.558863 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg\" (UniqueName: \"kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg\") pod \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " Oct 31 01:06:19.582399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.558937 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca\") pod \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " Oct 31 01:06:19.582399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.559005 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca\") pod \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\" (UID: \"5c6de869-5676-4e60-ac9b-dde7a51b7cd6\") " Oct 31 01:06:19.582399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.559242 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2hg\" (UniqueName: \"kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg\") pod \"payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.582399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.559536 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.586479 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.586334 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn"] Oct 31 01:06:19.586694 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.586665 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca" pod "5c6de869-5676-4e60-ac9b-dde7a51b7cd6" (UID: "5c6de869-5676-4e60-ac9b-dde7a51b7cd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:19.596348 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.593431 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg" pod "5c6de869-5676-4e60-ac9b-dde7a51b7cd6" (UID: "5c6de869-5676-4e60-ac9b-dde7a51b7cd6"). InnerVolumeSpecName "kube-api-access-g2cwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:19.602875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.602751 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692"] Oct 31 01:06:19.603614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.603595 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5c6de869-5676-4e60-ac9b-dde7a51b7cd6" containerName="revoke-keys" Oct 31 01:06:19.603676 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.603618 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6de869-5676-4e60-ac9b-dde7a51b7cd6" containerName="revoke-keys" Oct 31 01:06:19.603676 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.603654 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" Oct 31 01:06:19.603676 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.603662 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" Oct 31 01:06:19.603829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.603816 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e090d2-a606-4394-b9d4-a2f56bc255a8" containerName="istio-proxy" Oct 31 01:06:19.603879 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.603834 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6de869-5676-4e60-ac9b-dde7a51b7cd6" containerName="revoke-keys" Oct 31 01:06:19.606423 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.606116 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca" pod "5c6de869-5676-4e60-ac9b-dde7a51b7cd6" (UID: "5c6de869-5676-4e60-ac9b-dde7a51b7cd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:19.609091 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.609068 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.660656 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.660627 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2hg\" (UniqueName: \"kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg\") pod \"payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.660828 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.660805 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.660891 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.660875 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.660953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.660890 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2cwg\" (UniqueName: \"kubernetes.io/projected/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-kube-api-access-g2cwg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.660953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.660900 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6de869-5676-4e60-ac9b-dde7a51b7cd6-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.661580 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.661556 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.668066 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.668036 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692"] Oct 31 01:06:19.722732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.722698 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2hg\" (UniqueName: \"kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg\") pod \"payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.761986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.761949 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqw5b\" (UniqueName: \"kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b\") pod \"payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.762237 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.762214 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.776669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.776635 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:19.863429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.863386 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.863595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.863501 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqw5b\" (UniqueName: \"kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b\") pod \"payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.864330 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.864303 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.925477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.925444 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqw5b\" (UniqueName: \"kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b\") pod \"payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:19.940281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:19.940252 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:20.157701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.157672 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn"] Oct 31 01:06:20.158246 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:20.158214 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29680087_3751_4d74_8a11_c3364bbef202.slice/crio-c2d3994caf7cab8933445bd9e717ae8ad30e79d7bdd52c98cda78be548c440a7 WatchSource:0}: Error finding container c2d3994caf7cab8933445bd9e717ae8ad30e79d7bdd52c98cda78be548c440a7: Status 404 returned error can't find the container with id c2d3994caf7cab8933445bd9e717ae8ad30e79d7bdd52c98cda78be548c440a7 Oct 31 01:06:20.294011 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.293971 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" event={"ID":"29680087-3751-4d74-8a11-c3364bbef202","Type":"ContainerStarted","Data":"02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07"} Oct 31 01:06:20.294011 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.294037 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" event={"ID":"29680087-3751-4d74-8a11-c3364bbef202","Type":"ContainerStarted","Data":"c2d3994caf7cab8933445bd9e717ae8ad30e79d7bdd52c98cda78be548c440a7"} Oct 31 01:06:20.294548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.294210 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:20.296023 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.296001 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" Oct 31 01:06:20.296168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.296047 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm" event={"ID":"5c6de869-5676-4e60-ac9b-dde7a51b7cd6","Type":"ContainerDied","Data":"5fcf4a90e93d78807ac21156ba2669a99774daa697cf11da5a39d4e515a3408e"} Oct 31 01:06:20.296168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.296084 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fcf4a90e93d78807ac21156ba2669a99774daa697cf11da5a39d4e515a3408e" Oct 31 01:06:20.318916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.318895 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692"] Oct 31 01:06:20.320332 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:20.320302 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02086238_cbd8_4e28_a364_e2de71af6709.slice/crio-c365dec5fbe9fdd5f29a217bfa79031eaecded86dddbf6dc256cb532ad881396 WatchSource:0}: Error finding container c365dec5fbe9fdd5f29a217bfa79031eaecded86dddbf6dc256cb532ad881396: Status 404 returned error can't find the container with id c365dec5fbe9fdd5f29a217bfa79031eaecded86dddbf6dc256cb532ad881396 Oct 31 01:06:20.418243 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:20.418131 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" podStartSLOduration=1.418115771 podStartE2EDuration="1.418115771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:20.41603703 +0000 UTC m=+3760.509327242" watchObservedRunningTime="2026-10-31 01:06:20.418115771 +0000 UTC m=+3760.511405981" Oct 31 01:06:21.055511 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:21.055472 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw"] Oct 31 01:06:21.058490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:21.058459 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ed063f-967jw"] Oct 31 01:06:21.304312 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:21.304276 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" event={"ID":"02086238-cbd8-4e28-a364-e2de71af6709","Type":"ContainerStarted","Data":"b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4"} Oct 31 01:06:21.304755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:21.304342 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" event={"ID":"02086238-cbd8-4e28-a364-e2de71af6709","Type":"ContainerStarted","Data":"c365dec5fbe9fdd5f29a217bfa79031eaecded86dddbf6dc256cb532ad881396"} Oct 31 01:06:21.304755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:21.304372 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:21.422441 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:21.422325 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" podStartSLOduration=2.422309826 podStartE2EDuration="2.422309826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:21.420754665 +0000 UTC m=+3761.514044866" watchObservedRunningTime="2026-10-31 01:06:21.422309826 +0000 UTC m=+3761.515600056" Oct 31 01:06:22.041760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.041720 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1c04df-76b0-4854-99ac-fad88427c063" path="/var/lib/kubelet/pods/1f1c04df-76b0-4854-99ac-fad88427c063/volumes" Oct 31 01:06:22.328249 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.328127 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv"] Oct 31 01:06:22.332757 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.332736 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.436475 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.436434 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:22.436759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.436736 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:22.436850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.436749 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:06:22.436850 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.436815 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:22.441399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.441369 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv"] Oct 31 01:06:22.490621 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.490587 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:22.496351 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.496322 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.496492 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.496451 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gnq\" (UniqueName: \"kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.496572 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.496558 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.598250 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.598137 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.598440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.598290 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gnq\" (UniqueName: \"kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.598440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.598395 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.598917 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.598898 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.600926 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.600910 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.659450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.659421 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gnq\" (UniqueName: \"kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq\") pod \"maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:22.951124 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:22.951033 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:23.333953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:23.333923 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv"] Oct 31 01:06:23.337222 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:23.337164 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3f6794_05cf_4db9_9e64_a745e51d70b0.slice/crio-7405983fcc16ef46f0f11ae8ae836d0e15a04b0a54ab5315080aa5b6df8edc2b WatchSource:0}: Error finding container 7405983fcc16ef46f0f11ae8ae836d0e15a04b0a54ab5315080aa5b6df8edc2b: Status 404 returned error can't find the container with id 7405983fcc16ef46f0f11ae8ae836d0e15a04b0a54ab5315080aa5b6df8edc2b Oct 31 01:06:24.063322 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:24.063290 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs"] Oct 31 01:06:24.112151 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:24.112118 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7ebf049-jjrjs"] Oct 31 01:06:24.328242 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:24.328131 5943 generic.go:356] "Generic (PLEG): container finished" podID="6a3f6794-05cf-4db9-9e64-a745e51d70b0" containerID="c2ab84925101adbb6867cdd363284310b79e664aa3922ff5c9bc356a9374e37a" exitCode=0 Oct 31 01:06:24.328242 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:24.328216 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" event={"ID":"6a3f6794-05cf-4db9-9e64-a745e51d70b0","Type":"ContainerDied","Data":"c2ab84925101adbb6867cdd363284310b79e664aa3922ff5c9bc356a9374e37a"} Oct 31 01:06:24.328460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:24.328283 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" event={"ID":"6a3f6794-05cf-4db9-9e64-a745e51d70b0","Type":"ContainerStarted","Data":"7405983fcc16ef46f0f11ae8ae836d0e15a04b0a54ab5315080aa5b6df8edc2b"} Oct 31 01:06:24.565341 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:24.565312 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:06:25.485829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.485799 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:25.637840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.637756 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca\") pod \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " Oct 31 01:06:25.638258 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.637947 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca\") pod \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " Oct 31 01:06:25.638258 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.637979 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq\" (UniqueName: \"kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq\") pod \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\" (UID: \"6a3f6794-05cf-4db9-9e64-a745e51d70b0\") " Oct 31 01:06:25.639215 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.639160 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca" pod "6a3f6794-05cf-4db9-9e64-a745e51d70b0" (UID: "6a3f6794-05cf-4db9-9e64-a745e51d70b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:25.642655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.642613 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca" pod "6a3f6794-05cf-4db9-9e64-a745e51d70b0" (UID: "6a3f6794-05cf-4db9-9e64-a745e51d70b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:25.649397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.649360 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq" pod "6a3f6794-05cf-4db9-9e64-a745e51d70b0" (UID: "6a3f6794-05cf-4db9-9e64-a745e51d70b0"). InnerVolumeSpecName "kube-api-access-t9gnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:25.740332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.740292 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:25.740332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.740325 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gnq\" (UniqueName: \"kubernetes.io/projected/6a3f6794-05cf-4db9-9e64-a745e51d70b0-kube-api-access-t9gnq\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:25.740332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:25.740338 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3f6794-05cf-4db9-9e64-a745e51d70b0-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:26.044470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:26.044423 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7913dd62-b123-407c-9c9a-0edbf1198739" path="/var/lib/kubelet/pods/7913dd62-b123-407c-9c9a-0edbf1198739/volumes" Oct 31 01:06:26.344138 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:26.344053 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" Oct 31 01:06:26.344138 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:26.344083 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv" event={"ID":"6a3f6794-05cf-4db9-9e64-a745e51d70b0","Type":"ContainerDied","Data":"7405983fcc16ef46f0f11ae8ae836d0e15a04b0a54ab5315080aa5b6df8edc2b"} Oct 31 01:06:26.344138 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:26.344131 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7405983fcc16ef46f0f11ae8ae836d0e15a04b0a54ab5315080aa5b6df8edc2b" Oct 31 01:06:27.462703 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:27.462670 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:27.479133 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:27.478166 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:06:28.356077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:28.356044 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:28.554230 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:28.554158 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:31.308227 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:31.308168 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:06:32.053454 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:32.053425 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9"] Oct 31 01:06:32.103083 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:32.103047 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e78fb6a-gzlv9"] Oct 31 01:06:32.274486 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:32.274449 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:32.316931 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:32.316826 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:06:33.813827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:33.813790 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj"] Oct 31 01:06:34.045295 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:34.045254 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccadd53-f91c-4624-9f43-ca8abc917b38" path="/var/lib/kubelet/pods/7ccadd53-f91c-4624-9f43-ca8abc917b38/volumes" Oct 31 01:06:34.180392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:34.180298 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:34.403310 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:34.403275 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:35.048261 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:35.048168 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs"] Oct 31 01:06:35.051168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:35.051146 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98009d16-2vhfs"] Oct 31 01:06:36.042133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:36.042096 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c5a7de-8433-4e55-a66f-8a743f075af1" path="/var/lib/kubelet/pods/40c5a7de-8433-4e55-a66f-8a743f075af1/volumes" Oct 31 01:06:36.466916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:36.466808 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:37.629100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:37.629061 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:38.030084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:38.030046 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:38.294908 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:38.294818 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:38.514007 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:38.513974 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw"] Oct 31 01:06:38.563482 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:38.563399 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cc6150-dx4vw"] Oct 31 01:06:38.814539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:38.814446 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" podUID="83b1e695-efde-4463-80c7-0cd919b0829f" containerName="payload-processing" containerID="cri-o://62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32" gracePeriod=25 Oct 31 01:06:38.965994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:38.965964 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:39.154998 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.154965 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:06:39.243143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.243100 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume\") pod \"83b1e695-efde-4463-80c7-0cd919b0829f\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " Oct 31 01:06:39.243396 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.243173 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn\" (UniqueName: \"kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn\") pod \"83b1e695-efde-4463-80c7-0cd919b0829f\" (UID: \"83b1e695-efde-4463-80c7-0cd919b0829f\") " Oct 31 01:06:39.243601 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.243573 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume" pod "83b1e695-efde-4463-80c7-0cd919b0829f" (UID: "83b1e695-efde-4463-80c7-0cd919b0829f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:39.244047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.244029 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b1e695-efde-4463-80c7-0cd919b0829f-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:39.245718 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.245694 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn" pod "83b1e695-efde-4463-80c7-0cd919b0829f" (UID: "83b1e695-efde-4463-80c7-0cd919b0829f"). InnerVolumeSpecName "kube-api-access-7lwhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:39.281695 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.281660 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:39.345910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.345879 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwhn\" (UniqueName: \"kubernetes.io/projected/83b1e695-efde-4463-80c7-0cd919b0829f-kube-api-access-7lwhn\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:39.439102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.439011 5943 generic.go:356] "Generic (PLEG): container finished" podID="83b1e695-efde-4463-80c7-0cd919b0829f" containerID="62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32" exitCode=0 Oct 31 01:06:39.439102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.439067 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" Oct 31 01:06:39.439386 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.439093 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" event={"ID":"83b1e695-efde-4463-80c7-0cd919b0829f","Type":"ContainerDied","Data":"62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32"} Oct 31 01:06:39.439386 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.439157 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj" event={"ID":"83b1e695-efde-4463-80c7-0cd919b0829f","Type":"ContainerDied","Data":"43fd0d15a7c979c055d4d21976c0953d0a051baca79297090ca90e035fa91d48"} Oct 31 01:06:39.439386 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.439205 5943 scope.go:122] "RemoveContainer" containerID="62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32" Oct 31 01:06:39.450765 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.450741 5943 scope.go:122] "RemoveContainer" containerID="62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32" Oct 31 01:06:39.451142 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:39.451117 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32\": container with ID starting with 62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32 not found: ID does not exist" containerID="62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32" Oct 31 01:06:39.451248 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.451154 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32"} err="failed to get container status \"62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32\": rpc error: code = NotFound desc = could not find container \"62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32\": container with ID starting with 62825bf085adc174827430b91d6136eb333879082fc198e9696370a369f0de32 not found: ID does not exist" Oct 31 01:06:39.605549 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.605511 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj"] Oct 31 01:06:39.607776 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.607741 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:39.612818 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.612792 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fae60c-7f4bb58c89-thwjj"] Oct 31 01:06:39.759343 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.759314 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:39.872778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.872745 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv"] Oct 31 01:06:39.880213 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:39.877728 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fae60c-mj5pv"] Oct 31 01:06:40.042009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.041930 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3f6794-05cf-4db9-9e64-a745e51d70b0" path="/var/lib/kubelet/pods/6a3f6794-05cf-4db9-9e64-a745e51d70b0/volumes" Oct 31 01:06:40.042439 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.042422 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b1e695-efde-4463-80c7-0cd919b0829f" path="/var/lib/kubelet/pods/83b1e695-efde-4463-80c7-0cd919b0829f/volumes" Oct 31 01:06:40.042813 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.042800 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb70792f-2456-40e7-b5ea-4d6b0c31cbcf" path="/var/lib/kubelet/pods/eb70792f-2456-40e7-b5ea-4d6b0c31cbcf/volumes" Oct 31 01:06:40.320515 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.320421 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:40.450785 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.450750 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb"] Oct 31 01:06:40.451380 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.451360 5943 cpu_manager.go:414] "containerMap: removing container" podUID="6a3f6794-05cf-4db9-9e64-a745e51d70b0" containerName="revoke-keys" Oct 31 01:06:40.451483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.451383 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3f6794-05cf-4db9-9e64-a745e51d70b0" containerName="revoke-keys" Oct 31 01:06:40.451483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.451412 5943 cpu_manager.go:414] "containerMap: removing container" podUID="83b1e695-efde-4463-80c7-0cd919b0829f" containerName="payload-processing" Oct 31 01:06:40.451483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.451419 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b1e695-efde-4463-80c7-0cd919b0829f" containerName="payload-processing" Oct 31 01:06:40.451625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.451579 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3f6794-05cf-4db9-9e64-a745e51d70b0" containerName="revoke-keys" Oct 31 01:06:40.451625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.451597 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b1e695-efde-4463-80c7-0cd919b0829f" containerName="payload-processing" Oct 31 01:06:40.456584 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.456562 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.509423 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.509392 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:40.510021 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.509997 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:40.510206 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.510107 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:40.513865 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.513836 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb"] Oct 31 01:06:40.559293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.559269 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:06:40.561910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.561888 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.561979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.561962 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.562041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.562026 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tpb\" (UniqueName: \"kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.662979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.662885 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.663142 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.662988 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.663142 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.663027 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tpb\" (UniqueName: \"kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.663683 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.663657 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.665671 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.665644 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.724027 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.723996 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tpb\" (UniqueName: \"kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb\") pod \"maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:40.770625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:40.770593 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:41.152109 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:41.152077 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb"] Oct 31 01:06:41.152515 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:41.152189 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd888fa_4fff_4b4c_abcc_066bcfcc3424.slice/crio-76fc5f02ec2e46f7e574bb02956b074071ecbbf0a56d6a832b8c2b4aa5713515 WatchSource:0}: Error finding container 76fc5f02ec2e46f7e574bb02956b074071ecbbf0a56d6a832b8c2b4aa5713515: Status 404 returned error can't find the container with id 76fc5f02ec2e46f7e574bb02956b074071ecbbf0a56d6a832b8c2b4aa5713515 Oct 31 01:06:41.456993 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:41.456960 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" event={"ID":"bfd888fa-4fff-4b4c-abcc-066bcfcc3424","Type":"ContainerStarted","Data":"0b2facde85b287e121208b6458ebe1373b2a11ca79ad8b62792fbbd7b3af0cfe"} Oct 31 01:06:41.457146 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:41.457022 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" event={"ID":"bfd888fa-4fff-4b4c-abcc-066bcfcc3424","Type":"ContainerStarted","Data":"76fc5f02ec2e46f7e574bb02956b074071ecbbf0a56d6a832b8c2b4aa5713515"} Oct 31 01:06:41.574283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:41.574227 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" podStartSLOduration=1.574211239 podStartE2EDuration="1.574211239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:41.571595982 +0000 UTC m=+3781.664886190" watchObservedRunningTime="2026-10-31 01:06:41.574211239 +0000 UTC m=+3781.667501426" Oct 31 01:06:42.465561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:42.465475 5943 generic.go:356] "Generic (PLEG): container finished" podID="bfd888fa-4fff-4b4c-abcc-066bcfcc3424" containerID="0b2facde85b287e121208b6458ebe1373b2a11ca79ad8b62792fbbd7b3af0cfe" exitCode=0 Oct 31 01:06:42.465561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:42.465543 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" event={"ID":"bfd888fa-4fff-4b4c-abcc-066bcfcc3424","Type":"ContainerDied","Data":"0b2facde85b287e121208b6458ebe1373b2a11ca79ad8b62792fbbd7b3af0cfe"} Oct 31 01:06:43.048261 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.048221 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx"] Oct 31 01:06:43.050653 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.050625 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6d3a743-tq4vx"] Oct 31 01:06:43.633169 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.633139 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:43.801804 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.801762 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca\") pod \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " Oct 31 01:06:43.802026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.801923 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca\") pod \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " Oct 31 01:06:43.802026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.801998 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb\" (UniqueName: \"kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb\") pod \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\" (UID: \"bfd888fa-4fff-4b4c-abcc-066bcfcc3424\") " Oct 31 01:06:43.802289 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.802258 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca" pod "bfd888fa-4fff-4b4c-abcc-066bcfcc3424" (UID: "bfd888fa-4fff-4b4c-abcc-066bcfcc3424"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:43.802486 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.802468 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:43.804253 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.804224 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca" pod "bfd888fa-4fff-4b4c-abcc-066bcfcc3424" (UID: "bfd888fa-4fff-4b4c-abcc-066bcfcc3424"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:43.804357 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.804312 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb" pod "bfd888fa-4fff-4b4c-abcc-066bcfcc3424" (UID: "bfd888fa-4fff-4b4c-abcc-066bcfcc3424"). InnerVolumeSpecName "kube-api-access-c9tpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:43.903742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.903705 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:43.903742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:43.903736 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tpb\" (UniqueName: \"kubernetes.io/projected/bfd888fa-4fff-4b4c-abcc-066bcfcc3424-kube-api-access-c9tpb\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:44.041847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:44.041801 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7f3c13-5a9b-4cd1-9485-06a41fe960a0" path="/var/lib/kubelet/pods/ef7f3c13-5a9b-4cd1-9485-06a41fe960a0/volumes" Oct 31 01:06:44.480717 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:44.480685 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" Oct 31 01:06:44.480928 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:44.480717 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb" event={"ID":"bfd888fa-4fff-4b4c-abcc-066bcfcc3424","Type":"ContainerDied","Data":"76fc5f02ec2e46f7e574bb02956b074071ecbbf0a56d6a832b8c2b4aa5713515"} Oct 31 01:06:44.480928 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:44.480771 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76fc5f02ec2e46f7e574bb02956b074071ecbbf0a56d6a832b8c2b4aa5713515" Oct 31 01:06:44.892737 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:44.892699 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:45.092808 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:45.092772 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:45.421672 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:45.421639 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg"] Oct 31 01:06:45.422008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:45.421807 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:45.422146 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:45.422117 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" containerID="cri-o://cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06" gracePeriod=30 Oct 31 01:06:46.547348 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:46.547313 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:46.693278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:46.693171 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:46.900246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:46.900091 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr"] Oct 31 01:06:46.900431 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:46.900401 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" containerID="cri-o://0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6" gracePeriod=30 Oct 31 01:06:46.921794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:46.921759 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:47.641677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.641644 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26"] Oct 31 01:06:47.642265 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.642249 5943 cpu_manager.go:414] "containerMap: removing container" podUID="bfd888fa-4fff-4b4c-abcc-066bcfcc3424" containerName="revoke-keys" Oct 31 01:06:47.642315 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.642266 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd888fa-4fff-4b4c-abcc-066bcfcc3424" containerName="revoke-keys" Oct 31 01:06:47.642396 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.642387 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd888fa-4fff-4b4c-abcc-066bcfcc3424" containerName="revoke-keys" Oct 31 01:06:47.646899 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.646881 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.750262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.750230 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:47.750262 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.750252 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:47.750490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.750237 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:47.750490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.750310 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:06:47.752873 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.752851 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26"] Oct 31 01:06:47.755461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.755432 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.755618 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.755518 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.755618 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.755615 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqjm\" (UniqueName: \"kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.856461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.856419 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.856677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.856543 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqjm\" (UniqueName: \"kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.856677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.856607 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.857232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.857206 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.858959 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.858932 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.917272 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.917139 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqjm\" (UniqueName: \"kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:47.961059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:47.961024 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:48.345340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:48.345307 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26"] Oct 31 01:06:48.346444 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:48.346414 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b2c5bd_0fcc_4d21_8fa3_62144f983e3d.slice/crio-937428a8091ff5988c8aad2705724ae11feef4d72d56fa3d321fad988a305845 WatchSource:0}: Error finding container 937428a8091ff5988c8aad2705724ae11feef4d72d56fa3d321fad988a305845: Status 404 returned error can't find the container with id 937428a8091ff5988c8aad2705724ae11feef4d72d56fa3d321fad988a305845 Oct 31 01:06:48.510315 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:48.510284 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" event={"ID":"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d","Type":"ContainerStarted","Data":"937428a8091ff5988c8aad2705724ae11feef4d72d56fa3d321fad988a305845"} Oct 31 01:06:49.228307 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.228280 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw"] Oct 31 01:06:49.232348 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.232327 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.274232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.274172 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.274389 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.274361 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28g2\" (UniqueName: \"kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.274447 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.274398 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.291246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.291218 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw"] Oct 31 01:06:49.375452 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.375419 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28g2\" (UniqueName: \"kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.375452 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.375460 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.375730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.375556 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.376233 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.376205 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.378209 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.378164 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.436759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.436726 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28g2\" (UniqueName: \"kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.518232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.518201 5943 generic.go:356] "Generic (PLEG): container finished" podID="31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" containerID="e55e6c97630e6a5f9a9ace2d1cae8a39dee78966dd6ebb18a5d4f3ae30123019" exitCode=0 Oct 31 01:06:49.518410 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.518255 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" event={"ID":"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d","Type":"ContainerDied","Data":"e55e6c97630e6a5f9a9ace2d1cae8a39dee78966dd6ebb18a5d4f3ae30123019"} Oct 31 01:06:49.605699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.605670 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:49.985059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:49.985020 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw"] Oct 31 01:06:49.986373 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:06:49.986343 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d61b0c_ea8c_44be_9c72_5ffdd3cd5ae6.slice/crio-d76ef1324c2ea4d0993e77ce6ae58b8ba1d37795d8995d5b4b6df2e0d8ab3384 WatchSource:0}: Error finding container d76ef1324c2ea4d0993e77ce6ae58b8ba1d37795d8995d5b4b6df2e0d8ab3384: Status 404 returned error can't find the container with id d76ef1324c2ea4d0993e77ce6ae58b8ba1d37795d8995d5b4b6df2e0d8ab3384 Oct 31 01:06:50.264887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.264857 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:50.533137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.532974 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" event={"ID":"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6","Type":"ContainerStarted","Data":"20f563fe2ff5fdfb71d6364999130c4eff9c4b4210b750a66f299086332f858f"} Oct 31 01:06:50.533137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.533084 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" event={"ID":"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6","Type":"ContainerStarted","Data":"d76ef1324c2ea4d0993e77ce6ae58b8ba1d37795d8995d5b4b6df2e0d8ab3384"} Oct 31 01:06:50.651251 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.651161 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" podStartSLOduration=1.651144927 podStartE2EDuration="1.651144927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:50.650267492 +0000 UTC m=+3790.743557706" watchObservedRunningTime="2026-10-31 01:06:50.651144927 +0000 UTC m=+3790.744435113" Oct 31 01:06:50.711545 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.711523 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:50.719196 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.719159 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:06:50.791641 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791559 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.791826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791663 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.791826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791717 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.791826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791744 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.791826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791767 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm\" (UniqueName: \"kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm\") pod \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " Oct 31 01:06:50.791826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791803 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791842 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791894 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791935 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.791963 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk\") pod \"70fd23e6-8479-4811-aa12-3eb2a8521a28\" (UID: \"70fd23e6-8479-4811-aa12-3eb2a8521a28\") " Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792001 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca\") pod \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792002 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792045 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:50.792115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792074 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca\") pod \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\" (UID: \"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d\") " Oct 31 01:06:50.792561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792300 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:50.792561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792319 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:50.792561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792427 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:50.793047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792893 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.793047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792927 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fd23e6-8479-4811-aa12-3eb2a8521a28-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.793047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792943 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.793047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792959 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.793047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.792973 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.793047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.793033 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca" pod "31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" (UID: "31b2c5bd-0fcc-4d21-8fa3-62144f983e3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:50.794759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.794721 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:50.794888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.794812 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca" pod "31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" (UID: "31b2c5bd-0fcc-4d21-8fa3-62144f983e3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:50.794888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.794860 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm" pod "31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" (UID: "31b2c5bd-0fcc-4d21-8fa3-62144f983e3d"). InnerVolumeSpecName "kube-api-access-xwqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:50.794990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.794891 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:50.795029 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.794998 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:50.795768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.795750 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk" pod "70fd23e6-8479-4811-aa12-3eb2a8521a28" (UID: "70fd23e6-8479-4811-aa12-3eb2a8521a28"). InnerVolumeSpecName "kube-api-access-5whsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:50.893774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893743 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whsk\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-kube-api-access-5whsk\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.893774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893771 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.893981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893781 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.893981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893792 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.893981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893801 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqjm\" (UniqueName: \"kubernetes.io/projected/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d-kube-api-access-xwqjm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.893981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893810 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:50.893981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:50.893818 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fd23e6-8479-4811-aa12-3eb2a8521a28-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:51.540018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.539976 5943 generic.go:356] "Generic (PLEG): container finished" podID="88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" containerID="20f563fe2ff5fdfb71d6364999130c4eff9c4b4210b750a66f299086332f858f" exitCode=0 Oct 31 01:06:51.540551 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.540097 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" event={"ID":"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6","Type":"ContainerDied","Data":"20f563fe2ff5fdfb71d6364999130c4eff9c4b4210b750a66f299086332f858f"} Oct 31 01:06:51.541882 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.541862 5943 generic.go:356] "Generic (PLEG): container finished" podID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerID="cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06" exitCode=0 Oct 31 01:06:51.542008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.541918 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" Oct 31 01:06:51.542008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.541941 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" event={"ID":"70fd23e6-8479-4811-aa12-3eb2a8521a28","Type":"ContainerDied","Data":"cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06"} Oct 31 01:06:51.542008 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.541987 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg" event={"ID":"70fd23e6-8479-4811-aa12-3eb2a8521a28","Type":"ContainerDied","Data":"5020ecf3be16c92fc6f45f8d680ef1d08cdb621c451604d11574b4c13f4bebd1"} Oct 31 01:06:51.542164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.542018 5943 scope.go:122] "RemoveContainer" containerID="cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06" Oct 31 01:06:51.543481 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.543463 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" Oct 31 01:06:51.543595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.543478 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26" event={"ID":"31b2c5bd-0fcc-4d21-8fa3-62144f983e3d","Type":"ContainerDied","Data":"937428a8091ff5988c8aad2705724ae11feef4d72d56fa3d321fad988a305845"} Oct 31 01:06:51.543595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.543512 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="937428a8091ff5988c8aad2705724ae11feef4d72d56fa3d321fad988a305845" Oct 31 01:06:51.552972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.552946 5943 scope.go:122] "RemoveContainer" containerID="e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d" Oct 31 01:06:51.562794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.562777 5943 scope.go:122] "RemoveContainer" containerID="cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06" Oct 31 01:06:51.563048 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:51.563028 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06\": container with ID starting with cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06 not found: ID does not exist" containerID="cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06" Oct 31 01:06:51.563096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.563056 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06"} err="failed to get container status \"cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06\": rpc error: code = NotFound desc = could not find container \"cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06\": container with ID starting with cebdc1127a3a5b168f8fc74656a2f39b7d5c2868ba5fec426dfdccc55e8a0b06 not found: ID does not exist" Oct 31 01:06:51.563096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.563072 5943 scope.go:122] "RemoveContainer" containerID="e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d" Oct 31 01:06:51.563301 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:51.563282 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d\": container with ID starting with e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d not found: ID does not exist" containerID="e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d" Oct 31 01:06:51.563352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.563307 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d"} err="failed to get container status \"e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d\": rpc error: code = NotFound desc = could not find container \"e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d\": container with ID starting with e442fd28ae953db3994bde226956e0fa5ea3383eeb50753b63a81671bac7594d not found: ID does not exist" Oct 31 01:06:51.821062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.820973 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg"] Oct 31 01:06:51.870827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:51.870784 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cc6150-openshift-default-65645f785-rx5bg"] Oct 31 01:06:52.041997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.041966 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" path="/var/lib/kubelet/pods/70fd23e6-8479-4811-aa12-3eb2a8521a28/volumes" Oct 31 01:06:52.171401 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.171367 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:06:52.209597 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209564 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.209778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209613 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.209778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209694 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.209778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209756 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.209970 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209804 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.209970 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209858 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.209970 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209883 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:52.209970 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.209898 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.210216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210004 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.210216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210026 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:52.210216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210052 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert\") pod \"1875df41-6231-4298-9709-480d1fce66fa\" (UID: \"1875df41-6231-4298-9709-480d1fce66fa\") " Oct 31 01:06:52.210216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210063 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:52.210216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210197 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:52.210499 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210432 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:52.210861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210835 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.210994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210869 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.210994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210883 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.210994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210896 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875df41-6231-4298-9709-480d1fce66fa-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.210994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.210909 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.212105 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.212085 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:52.212360 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.212338 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "kube-api-access-7vhvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:52.212593 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.212575 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:52.212666 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.212591 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy" pod "1875df41-6231-4298-9709-480d1fce66fa" (UID: "1875df41-6231-4298-9709-480d1fce66fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:52.312193 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.312158 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vhvz\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-kube-api-access-7vhvz\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.312505 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.312216 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875df41-6231-4298-9709-480d1fce66fa-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.312505 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.312232 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875df41-6231-4298-9709-480d1fce66fa-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.312505 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.312244 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875df41-6231-4298-9709-480d1fce66fa-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.553210 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.553162 5943 generic.go:356] "Generic (PLEG): container finished" podID="1875df41-6231-4298-9709-480d1fce66fa" containerID="0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6" exitCode=0 Oct 31 01:06:52.553702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.553210 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" event={"ID":"1875df41-6231-4298-9709-480d1fce66fa","Type":"ContainerDied","Data":"0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6"} Oct 31 01:06:52.553702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.553245 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" Oct 31 01:06:52.553702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.553275 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr" event={"ID":"1875df41-6231-4298-9709-480d1fce66fa","Type":"ContainerDied","Data":"397091fff28a27cd4bd63ad795e8ffd27e657bc9e9b0bfdae7c693b55cfb3d40"} Oct 31 01:06:52.553702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.553306 5943 scope.go:122] "RemoveContainer" containerID="0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6" Oct 31 01:06:52.564513 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.564496 5943 scope.go:122] "RemoveContainer" containerID="a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879" Oct 31 01:06:52.575106 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.575086 5943 scope.go:122] "RemoveContainer" containerID="0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6" Oct 31 01:06:52.575415 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:52.575391 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6\": container with ID starting with 0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6 not found: ID does not exist" containerID="0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6" Oct 31 01:06:52.575499 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.575428 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6"} err="failed to get container status \"0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6\": rpc error: code = NotFound desc = could not find container \"0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6\": container with ID starting with 0ffb70e3b133ff0edec25bb4fe83d09871a07177b061bf56d2ae2d4fcf02e7b6 not found: ID does not exist" Oct 31 01:06:52.575499 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.575453 5943 scope.go:122] "RemoveContainer" containerID="a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879" Oct 31 01:06:52.575730 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:52.575712 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879\": container with ID starting with a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879 not found: ID does not exist" containerID="a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879" Oct 31 01:06:52.575773 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.575738 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879"} err="failed to get container status \"a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879\": rpc error: code = NotFound desc = could not find container \"a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879\": container with ID starting with a9938138ce4b85d894d26d0c14bf3519d577b2e28371f64df0d0e81b14d0c879 not found: ID does not exist" Oct 31 01:06:52.687367 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.687339 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:52.709559 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.709531 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np"] Oct 31 01:06:52.718627 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.718600 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2\" (UniqueName: \"kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2\") pod \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " Oct 31 01:06:52.718804 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.718681 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca\") pod \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " Oct 31 01:06:52.718804 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.718742 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca\") pod \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\" (UID: \"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6\") " Oct 31 01:06:52.719361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.719334 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr"] Oct 31 01:06:52.719361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.719372 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca" pod "88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" (UID: "88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:52.721116 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.721088 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2" pod "88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" (UID: "88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6"). InnerVolumeSpecName "kube-api-access-m28g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:52.721250 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.721148 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca" pod "88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" (UID: "88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:52.726359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.726337 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fae60c-openshift-default-68b4895869-scwnr"] Oct 31 01:06:52.820690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.820603 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m28g2\" (UniqueName: \"kubernetes.io/projected/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-kube-api-access-m28g2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.820690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.820634 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.820690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.820650 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:52.969148 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:52.969119 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:53.114369 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:53.114282 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:53.564283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:53.564253 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" Oct 31 01:06:53.564283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:53.564281 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw" event={"ID":"88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6","Type":"ContainerDied","Data":"d76ef1324c2ea4d0993e77ce6ae58b8ba1d37795d8995d5b4b6df2e0d8ab3384"} Oct 31 01:06:53.564858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:53.564331 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d76ef1324c2ea4d0993e77ce6ae58b8ba1d37795d8995d5b4b6df2e0d8ab3384" Oct 31 01:06:54.067202 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:54.066343 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1875df41-6231-4298-9709-480d1fce66fa" path="/var/lib/kubelet/pods/1875df41-6231-4298-9709-480d1fce66fa/volumes" Oct 31 01:06:54.079570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:54.073294 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd"] Oct 31 01:06:54.079570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:54.075468 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fd82-wwbtd"] Oct 31 01:06:55.259123 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:55.259079 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:56.044380 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:56.044345 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17889338-f43a-4df9-b0f4-077d82386c39" path="/var/lib/kubelet/pods/17889338-f43a-4df9-b0f4-077d82386c39/volumes" Oct 31 01:06:57.477037 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:57.476996 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:06:57.710861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:57.710815 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" podUID="2793def7-6e89-412b-b90c-e803f6911f76" containerName="payload-pre-processing" containerID="cri-o://9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c" gracePeriod=25 Oct 31 01:06:57.988101 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:57.988041 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:06:58.095860 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.095820 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh\") pod \"2793def7-6e89-412b-b90c-e803f6911f76\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " Oct 31 01:06:58.096128 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.096108 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume\") pod \"2793def7-6e89-412b-b90c-e803f6911f76\" (UID: \"2793def7-6e89-412b-b90c-e803f6911f76\") " Oct 31 01:06:58.096546 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.096525 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume" pod "2793def7-6e89-412b-b90c-e803f6911f76" (UID: "2793def7-6e89-412b-b90c-e803f6911f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:58.096796 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.096735 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2793def7-6e89-412b-b90c-e803f6911f76-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:58.098147 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.098118 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh" pod "2793def7-6e89-412b-b90c-e803f6911f76" (UID: "2793def7-6e89-412b-b90c-e803f6911f76"). InnerVolumeSpecName "kube-api-access-6slbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:58.198571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.198540 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/2793def7-6e89-412b-b90c-e803f6911f76-kube-api-access-6slbh\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:58.608058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.608017 5943 generic.go:356] "Generic (PLEG): container finished" podID="2793def7-6e89-412b-b90c-e803f6911f76" containerID="9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c" exitCode=0 Oct 31 01:06:58.608526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.608085 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" Oct 31 01:06:58.608526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.608101 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" event={"ID":"2793def7-6e89-412b-b90c-e803f6911f76","Type":"ContainerDied","Data":"9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c"} Oct 31 01:06:58.608526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.608225 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np" event={"ID":"2793def7-6e89-412b-b90c-e803f6911f76","Type":"ContainerDied","Data":"7f7794aa6a642dde87ed7f6395e3f81a307b0c7c3de0bddef3c68441c229235e"} Oct 31 01:06:58.608526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.608256 5943 scope.go:122] "RemoveContainer" containerID="9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c" Oct 31 01:06:58.619221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.619173 5943 scope.go:122] "RemoveContainer" containerID="9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c" Oct 31 01:06:58.619531 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:06:58.619511 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c\": container with ID starting with 9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c not found: ID does not exist" containerID="9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c" Oct 31 01:06:58.619606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.619540 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c"} err="failed to get container status \"9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c\": rpc error: code = NotFound desc = could not find container \"9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c\": container with ID starting with 9f227a8ed45132bcdd5d8e0a13e8671c020696f6f4f20c271a0253741914be6c not found: ID does not exist" Oct 31 01:06:58.773306 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.773273 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np"] Oct 31 01:06:58.779419 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:58.779395 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaa842-68b87c5b59-vf5np"] Oct 31 01:06:59.566111 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:59.566076 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:59.737988 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:06:59.737948 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:00.041573 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:00.041537 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2793def7-6e89-412b-b90c-e803f6911f76" path="/var/lib/kubelet/pods/2793def7-6e89-412b-b90c-e803f6911f76/volumes" Oct 31 01:07:00.045703 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:00.045680 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:00.190033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:00.189990 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:02.074222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:02.074169 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:02.269049 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:02.268982 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:02.525883 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:02.525851 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:07:04.843827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:04.843792 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:09.846427 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:09.846388 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:14.005300 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:14.005264 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:15.503615 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:15.503565 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:15.724431 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:15.724394 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:15.868315 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:15.868281 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:16.015146 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:16.015113 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm"] Oct 31 01:07:16.064318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:16.064276 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-16b984-q82nm"] Oct 31 01:07:16.433612 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:16.433579 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:18.041363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:18.041331 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6de869-5676-4e60-ac9b-dde7a51b7cd6" path="/var/lib/kubelet/pods/5c6de869-5676-4e60-ac9b-dde7a51b7cd6/volumes" Oct 31 01:07:20.878051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:20.878002 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:22.632386 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:22.632348 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:22.867955 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:22.867914 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:23.098929 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.098893 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:23.944741 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.944702 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg"] Oct 31 01:07:23.945484 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945463 5943 cpu_manager.go:414] "containerMap: removing container" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945486 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945529 5943 cpu_manager.go:414] "containerMap: removing container" podUID="31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" containerName="revoke-keys" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945537 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" containerName="revoke-keys" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945564 5943 cpu_manager.go:414] "containerMap: removing container" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945572 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945587 5943 cpu_manager.go:414] "containerMap: removing container" podUID="88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" containerName="revoke-keys" Oct 31 01:07:23.945606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945596 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" containerName="revoke-keys" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945624 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945632 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945649 5943 cpu_manager.go:414] "containerMap: removing container" podUID="2793def7-6e89-412b-b90c-e803f6911f76" containerName="payload-pre-processing" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945657 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2793def7-6e89-412b-b90c-e803f6911f76" containerName="payload-pre-processing" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945669 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945677 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945810 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" containerName="revoke-keys" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945838 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945849 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" containerName="revoke-keys" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945862 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945873 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="2793def7-6e89-412b-b90c-e803f6911f76" containerName="payload-pre-processing" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945881 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1875df41-6231-4298-9709-480d1fce66fa" containerName="istio-proxy" Oct 31 01:07:23.946004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.945894 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fd23e6-8479-4811-aa12-3eb2a8521a28" containerName="istio-proxy" Oct 31 01:07:23.950737 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:23.950709 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.035437 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.035401 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.035633 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.035607 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.035707 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.035634 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqbx\" (UniqueName: \"kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.053519 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.053485 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:24.053696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.053678 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:24.053777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.053692 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:24.053777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.053737 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:07:24.056026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.056004 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg"] Oct 31 01:07:24.137344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.137307 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.137576 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.137365 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqbx\" (UniqueName: \"kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.137657 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.137633 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.138354 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.138267 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.139931 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.139909 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.198927 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.198857 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqbx\" (UniqueName: \"kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.263975 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.263945 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:24.646040 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.646012 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg"] Oct 31 01:07:24.648038 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:07:24.648008 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66a1b51_f158_47e5_81a8_a0f446d39a44.slice/crio-03b2d124c240bd87e62ddae671730a9849621ecfd3e8ebedcb66fbb21a17abba WatchSource:0}: Error finding container 03b2d124c240bd87e62ddae671730a9849621ecfd3e8ebedcb66fbb21a17abba: Status 404 returned error can't find the container with id 03b2d124c240bd87e62ddae671730a9849621ecfd3e8ebedcb66fbb21a17abba Oct 31 01:07:24.812237 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:24.812193 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" event={"ID":"a66a1b51-f158-47e5-81a8-a0f446d39a44","Type":"ContainerStarted","Data":"03b2d124c240bd87e62ddae671730a9849621ecfd3e8ebedcb66fbb21a17abba"} Oct 31 01:07:25.820081 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:25.820045 5943 generic.go:356] "Generic (PLEG): container finished" podID="a66a1b51-f158-47e5-81a8-a0f446d39a44" containerID="a307b07f074e576d3eb0f986fd4183e7d1a0e3f863021acafef5d1ff8661c6ad" exitCode=0 Oct 31 01:07:25.820562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:25.820096 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" event={"ID":"a66a1b51-f158-47e5-81a8-a0f446d39a44","Type":"ContainerDied","Data":"a307b07f074e576d3eb0f986fd4183e7d1a0e3f863021acafef5d1ff8661c6ad"} Oct 31 01:07:26.181266 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:26.181144 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:26.980167 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:26.980143 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:27.072295 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.072262 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca\") pod \"a66a1b51-f158-47e5-81a8-a0f446d39a44\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " Oct 31 01:07:27.072468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.072393 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca\") pod \"a66a1b51-f158-47e5-81a8-a0f446d39a44\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " Oct 31 01:07:27.072468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.072434 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx\" (UniqueName: \"kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx\") pod \"a66a1b51-f158-47e5-81a8-a0f446d39a44\" (UID: \"a66a1b51-f158-47e5-81a8-a0f446d39a44\") " Oct 31 01:07:27.072686 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.072658 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca" pod "a66a1b51-f158-47e5-81a8-a0f446d39a44" (UID: "a66a1b51-f158-47e5-81a8-a0f446d39a44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:27.072897 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.072881 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:27.074683 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.074658 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx" pod "a66a1b51-f158-47e5-81a8-a0f446d39a44" (UID: "a66a1b51-f158-47e5-81a8-a0f446d39a44"). InnerVolumeSpecName "kube-api-access-xbqbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:27.074787 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.074735 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca" pod "a66a1b51-f158-47e5-81a8-a0f446d39a44" (UID: "a66a1b51-f158-47e5-81a8-a0f446d39a44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:27.174461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.174383 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66a1b51-f158-47e5-81a8-a0f446d39a44-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:27.174750 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.174727 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqbx\" (UniqueName: \"kubernetes.io/projected/a66a1b51-f158-47e5-81a8-a0f446d39a44-kube-api-access-xbqbx\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:27.477676 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:07:27.477583 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:07:27.834954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.834925 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" Oct 31 01:07:27.835139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.834961 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg" event={"ID":"a66a1b51-f158-47e5-81a8-a0f446d39a44","Type":"ContainerDied","Data":"03b2d124c240bd87e62ddae671730a9849621ecfd3e8ebedcb66fbb21a17abba"} Oct 31 01:07:27.835139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:27.835003 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b2d124c240bd87e62ddae671730a9849621ecfd3e8ebedcb66fbb21a17abba" Oct 31 01:07:31.204570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:31.204537 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:31.709987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:31.709943 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:35.525012 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:35.524972 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc"] Oct 31 01:07:35.779922 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:35.779829 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:35.923018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:35.922969 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:36.147023 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:36.146924 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:40.299203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:40.299134 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:40.526947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:40.526884 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerName="payload-pre-processing" containerID="cri-o://bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688" gracePeriod=25 Oct 31 01:07:41.857038 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:41.857008 5943 scope.go:122] "RemoveContainer" containerID="c8eefe9d65430ab2fe7f4fbeb09513746731ce9d191067d96893e387d841eba8" Oct 31 01:07:41.867851 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:41.867826 5943 scope.go:122] "RemoveContainer" containerID="d4b6258655b8e9fd603859e9a039460e7c83e9aeadb175d66c7927cc7ee583ed" Oct 31 01:07:41.873212 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:41.873153 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:41.880053 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:41.880026 5943 scope.go:122] "RemoveContainer" containerID="9904d3c083c80d95103941855cb02b44652090e569187eb1f51d540aaf33bb37" Oct 31 01:07:41.889260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:41.889239 5943 scope.go:122] "RemoveContainer" containerID="9823fa3067bccbeacaf38cc31ed88a3202b01a36e8f87a7fbe53f8db5c53ff64" Oct 31 01:07:41.897874 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:41.897856 5943 scope.go:122] "RemoveContainer" containerID="dc4c30abb231cca1f949f69d0d59f1cc90073af2c8e9820a406ec49dc9573dba" Oct 31 01:07:42.016047 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:42.016016 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:42.164432 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:42.164346 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:42.382916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:42.382871 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:42.464702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:42.464609 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw"] Oct 31 01:07:42.513478 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:42.513444 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a7eb09-rvhpw"] Oct 31 01:07:42.867060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:42.867023 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:44.043573 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:44.043538 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6" path="/var/lib/kubelet/pods/88d61b0c-ea8c-44be-9c72-5ffdd3cd5ae6/volumes" Oct 31 01:07:46.684445 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:46.684404 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:48.353663 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.353631 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:48.572071 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.572031 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:48.705605 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.705521 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:07:48.716290 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.716248 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:48.841252 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.841221 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26"] Oct 31 01:07:48.890951 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.890510 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f13553-v4f26"] Oct 31 01:07:48.925038 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:48.924997 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:49.070831 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:49.070774 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:49.288210 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:49.288137 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:49.442686 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:49.442596 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:50.041478 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.041446 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b2c5bd-0fcc-4d21-8fa3-62144f983e3d" path="/var/lib/kubelet/pods/31b2c5bd-0fcc-4d21-8fa3-62144f983e3d/volumes" Oct 31 01:07:50.162825 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.162792 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx"] Oct 31 01:07:50.163355 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.163340 5943 cpu_manager.go:414] "containerMap: removing container" podUID="a66a1b51-f158-47e5-81a8-a0f446d39a44" containerName="revoke-keys" Oct 31 01:07:50.163355 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.163355 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66a1b51-f158-47e5-81a8-a0f446d39a44" containerName="revoke-keys" Oct 31 01:07:50.163464 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.163455 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66a1b51-f158-47e5-81a8-a0f446d39a44" containerName="revoke-keys" Oct 31 01:07:50.166785 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.166770 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.219015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.218981 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:50.219015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.218978 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:50.219248 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.219079 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:50.268782 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.268753 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:07:50.271379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.271357 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx"] Oct 31 01:07:50.274464 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.274438 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2tq\" (UniqueName: \"kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.274571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.274555 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.274618 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.274594 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.376072 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.375983 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.376072 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.376033 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.376340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.376134 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2tq\" (UniqueName: \"kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.376807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.376786 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.378568 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.378544 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.437114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.437083 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2tq\" (UniqueName: \"kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.479847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.479811 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:50.855928 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:50.855895 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx"] Oct 31 01:07:50.858155 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:07:50.858126 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ee1761_9e3b_4cf3_b6de_c0b134f62e57.slice/crio-95c1e5daf12f7546dfe7eff7874364761224046c88355431ea0415c0476f68b5 WatchSource:0}: Error finding container 95c1e5daf12f7546dfe7eff7874364761224046c88355431ea0415c0476f68b5: Status 404 returned error can't find the container with id 95c1e5daf12f7546dfe7eff7874364761224046c88355431ea0415c0476f68b5 Oct 31 01:07:51.026667 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:51.026628 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" event={"ID":"18ee1761-9e3b-4cf3-b6de-c0b134f62e57","Type":"ContainerStarted","Data":"95c1e5daf12f7546dfe7eff7874364761224046c88355431ea0415c0476f68b5"} Oct 31 01:07:52.035464 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:52.035434 5943 generic.go:356] "Generic (PLEG): container finished" podID="18ee1761-9e3b-4cf3-b6de-c0b134f62e57" containerID="9569ae42f595475964c39a562482d9e8e6cfbebd3833c2309c53e8f190996513" exitCode=0 Oct 31 01:07:52.041239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:52.041211 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" event={"ID":"18ee1761-9e3b-4cf3-b6de-c0b134f62e57","Type":"ContainerDied","Data":"9569ae42f595475964c39a562482d9e8e6cfbebd3833c2309c53e8f190996513"} Oct 31 01:07:53.194562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.194537 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:53.310519 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.310430 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca\") pod \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " Oct 31 01:07:53.310715 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.310556 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq\" (UniqueName: \"kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq\") pod \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " Oct 31 01:07:53.310715 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.310694 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca\") pod \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\" (UID: \"18ee1761-9e3b-4cf3-b6de-c0b134f62e57\") " Oct 31 01:07:53.310854 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.310830 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca" pod "18ee1761-9e3b-4cf3-b6de-c0b134f62e57" (UID: "18ee1761-9e3b-4cf3-b6de-c0b134f62e57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:53.311143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.311122 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:53.313373 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.313344 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq" pod "18ee1761-9e3b-4cf3-b6de-c0b134f62e57" (UID: "18ee1761-9e3b-4cf3-b6de-c0b134f62e57"). InnerVolumeSpecName "kube-api-access-wx2tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:53.313520 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.313497 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca" pod "18ee1761-9e3b-4cf3-b6de-c0b134f62e57" (UID: "18ee1761-9e3b-4cf3-b6de-c0b134f62e57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:53.412530 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.412493 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2tq\" (UniqueName: \"kubernetes.io/projected/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-kube-api-access-wx2tq\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:53.412530 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.412528 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ee1761-9e3b-4cf3-b6de-c0b134f62e57-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:53.694333 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:53.694232 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:54.050956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:54.050928 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" Oct 31 01:07:54.051199 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:54.050963 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx" event={"ID":"18ee1761-9e3b-4cf3-b6de-c0b134f62e57","Type":"ContainerDied","Data":"95c1e5daf12f7546dfe7eff7874364761224046c88355431ea0415c0476f68b5"} Oct 31 01:07:54.051199 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:54.051018 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c1e5daf12f7546dfe7eff7874364761224046c88355431ea0415c0476f68b5" Oct 31 01:07:55.450374 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:55.450335 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:55.632674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:55.632638 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:55.788205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:55.788128 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:56.661538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.661503 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6"] Oct 31 01:07:56.662025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.662012 5943 cpu_manager.go:414] "containerMap: removing container" podUID="18ee1761-9e3b-4cf3-b6de-c0b134f62e57" containerName="revoke-keys" Oct 31 01:07:56.662073 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.662026 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ee1761-9e3b-4cf3-b6de-c0b134f62e57" containerName="revoke-keys" Oct 31 01:07:56.662157 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.662149 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ee1761-9e3b-4cf3-b6de-c0b134f62e57" containerName="revoke-keys" Oct 31 01:07:56.665673 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.665659 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.723875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.723837 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6"] Oct 31 01:07:56.754096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.754056 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgs8z\" (UniqueName: \"kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.754311 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.754137 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.754384 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.754364 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.769747 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.769707 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:56.769747 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.769730 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:56.769985 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.769861 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:56.769985 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.769917 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:07:56.855652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.855618 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.855851 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.855714 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgs8z\" (UniqueName: \"kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.855851 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.855773 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.856553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.856467 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.858320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.858298 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.916918 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.916845 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgs8z\" (UniqueName: \"kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:56.979102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:56.979053 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:07:57.360866 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:57.360843 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6"] Oct 31 01:07:57.362672 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:07:57.362644 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad935983_be86_4e2f_bf12_0aaa1bd8e716.slice/crio-7908054ddff1215f8e45086c3f7013ded6c1e2cc2cd16213c5006d105aaa03c8 WatchSource:0}: Error finding container 7908054ddff1215f8e45086c3f7013ded6c1e2cc2cd16213c5006d105aaa03c8: Status 404 returned error can't find the container with id 7908054ddff1215f8e45086c3f7013ded6c1e2cc2cd16213c5006d105aaa03c8 Oct 31 01:07:57.476785 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:07:57.476735 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:07:58.081015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:58.080978 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" event={"ID":"ad935983-be86-4e2f-bf12-0aaa1bd8e716","Type":"ContainerStarted","Data":"73f5e2811b132fc26da70ab4b95f2e42ae1dff793094214dad7a041e5c180507"} Oct 31 01:07:58.081453 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:58.081037 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" event={"ID":"ad935983-be86-4e2f-bf12-0aaa1bd8e716","Type":"ContainerStarted","Data":"7908054ddff1215f8e45086c3f7013ded6c1e2cc2cd16213c5006d105aaa03c8"} Oct 31 01:07:58.197891 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:58.197832 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" podStartSLOduration=2.197812463 podStartE2EDuration="2.197812463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:58.19555588 +0000 UTC m=+3858.288846090" watchObservedRunningTime="2026-10-31 01:07:58.197812463 +0000 UTC m=+3858.291102650" Oct 31 01:07:58.705003 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:58.704965 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:07:58.879915 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:58.879885 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:59.089672 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:59.089633 5943 generic.go:356] "Generic (PLEG): container finished" podID="ad935983-be86-4e2f-bf12-0aaa1bd8e716" containerID="73f5e2811b132fc26da70ab4b95f2e42ae1dff793094214dad7a041e5c180507" exitCode=0 Oct 31 01:07:59.090061 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:07:59.089699 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" event={"ID":"ad935983-be86-4e2f-bf12-0aaa1bd8e716","Type":"ContainerDied","Data":"73f5e2811b132fc26da70ab4b95f2e42ae1dff793094214dad7a041e5c180507"} Oct 31 01:08:00.235524 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.235495 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:08:00.396572 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.396491 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca\") pod \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " Oct 31 01:08:00.396714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.396659 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z\" (UniqueName: \"kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z\") pod \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " Oct 31 01:08:00.396714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.396692 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca\") pod \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\" (UID: \"ad935983-be86-4e2f-bf12-0aaa1bd8e716\") " Oct 31 01:08:00.396913 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.396884 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca" pod "ad935983-be86-4e2f-bf12-0aaa1bd8e716" (UID: "ad935983-be86-4e2f-bf12-0aaa1bd8e716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:00.397249 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.397210 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:00.399133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.399100 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca" pod "ad935983-be86-4e2f-bf12-0aaa1bd8e716" (UID: "ad935983-be86-4e2f-bf12-0aaa1bd8e716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:00.399352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.399222 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z" pod "ad935983-be86-4e2f-bf12-0aaa1bd8e716" (UID: "ad935983-be86-4e2f-bf12-0aaa1bd8e716"). InnerVolumeSpecName "kube-api-access-fgs8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:00.399352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.399314 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:00.497997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.497964 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgs8z\" (UniqueName: \"kubernetes.io/projected/ad935983-be86-4e2f-bf12-0aaa1bd8e716-kube-api-access-fgs8z\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:00.497997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.497994 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad935983-be86-4e2f-bf12-0aaa1bd8e716-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:00.762563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.762538 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 01:08:00.903051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.903020 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume\") pod \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " Oct 31 01:08:00.903249 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.903091 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf\" (UniqueName: \"kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf\") pod \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\" (UID: \"61de143f-81b8-4cd1-b5b6-5f5dd780c071\") " Oct 31 01:08:00.903470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.903440 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume" pod "61de143f-81b8-4cd1-b5b6-5f5dd780c071" (UID: "61de143f-81b8-4cd1-b5b6-5f5dd780c071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:00.903847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.903826 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61de143f-81b8-4cd1-b5b6-5f5dd780c071-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:00.905470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:00.905447 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf" pod "61de143f-81b8-4cd1-b5b6-5f5dd780c071" (UID: "61de143f-81b8-4cd1-b5b6-5f5dd780c071"). InnerVolumeSpecName "kube-api-access-xwjnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:01.004807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.004721 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjnf\" (UniqueName: \"kubernetes.io/projected/61de143f-81b8-4cd1-b5b6-5f5dd780c071-kube-api-access-xwjnf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.103105 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.103075 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" Oct 31 01:08:01.103300 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.103075 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6" event={"ID":"ad935983-be86-4e2f-bf12-0aaa1bd8e716","Type":"ContainerDied","Data":"7908054ddff1215f8e45086c3f7013ded6c1e2cc2cd16213c5006d105aaa03c8"} Oct 31 01:08:01.103300 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.103230 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7908054ddff1215f8e45086c3f7013ded6c1e2cc2cd16213c5006d105aaa03c8" Oct 31 01:08:01.104654 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.104626 5943 generic.go:356] "Generic (PLEG): container finished" podID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerID="bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688" exitCode=0 Oct 31 01:08:01.104805 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.104669 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" Oct 31 01:08:01.104805 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.104709 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" event={"ID":"61de143f-81b8-4cd1-b5b6-5f5dd780c071","Type":"ContainerDied","Data":"bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688"} Oct 31 01:08:01.104805 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.104758 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc" event={"ID":"61de143f-81b8-4cd1-b5b6-5f5dd780c071","Type":"ContainerDied","Data":"d1d396d1f928c3cc02f3bf39066e00fd385893c10ec5f3c61cdd27d1e001641e"} Oct 31 01:08:01.104805 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.104786 5943 scope.go:122] "RemoveContainer" containerID="bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688" Oct 31 01:08:01.116111 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.116095 5943 scope.go:122] "RemoveContainer" containerID="bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688" Oct 31 01:08:01.116400 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:08:01.116383 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688\": container with ID starting with bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688 not found: ID does not exist" containerID="bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688" Oct 31 01:08:01.116450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.116409 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688"} err="failed to get container status \"bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688\": rpc error: code = NotFound desc = could not find container \"bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688\": container with ID starting with bff98c352c21b64c8912a5852514daad14a01568d86c20fec894bb086b0aa688 not found: ID does not exist" Oct 31 01:08:01.320488 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.320458 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc"] Oct 31 01:08:01.325502 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:01.325471 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bf325c-56b9c45d8c-4pqfc"] Oct 31 01:08:02.041468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:02.041433 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" path="/var/lib/kubelet/pods/61de143f-81b8-4cd1-b5b6-5f5dd780c071/volumes" Oct 31 01:08:02.106887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:02.106854 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:02.255958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:02.255921 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:03.861855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:03.861801 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:08.158425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:08.158386 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:08.555649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:08.555619 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:08.704918 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:08.704880 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:09.397218 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:09.397172 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:08:09.665729 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:09.665641 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:09.853107 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:09.853075 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:10.006584 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:10.006550 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:10.178122 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:10.178082 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb"] Oct 31 01:08:10.225089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:10.225057 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-aaa842-tk8bb"] Oct 31 01:08:10.614897 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:10.614854 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:12.060528 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:12.060493 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd888fa-4fff-4b4c-abcc-066bcfcc3424" path="/var/lib/kubelet/pods/bfd888fa-4fff-4b4c-abcc-066bcfcc3424/volumes" Oct 31 01:08:14.283944 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:14.283911 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:15.897663 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:15.897623 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.146107 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:16.146009 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.292530 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:16.292493 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.362103 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:16.362069 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6"] Oct 31 01:08:16.412634 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:16.412553 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e1a6c-mlth6"] Oct 31 01:08:16.808914 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:16.808879 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.982206 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:16.982140 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:17.296384 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:17.296349 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:18.042460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.042422 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad935983-be86-4e2f-bf12-0aaa1bd8e716" path="/var/lib/kubelet/pods/ad935983-be86-4e2f-bf12-0aaa1bd8e716/volumes" Oct 31 01:08:18.416084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.415991 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr"] Oct 31 01:08:18.416616 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.416600 5943 cpu_manager.go:414] "containerMap: removing container" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerName="payload-pre-processing" Oct 31 01:08:18.416665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.416618 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerName="payload-pre-processing" Oct 31 01:08:18.416665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.416651 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ad935983-be86-4e2f-bf12-0aaa1bd8e716" containerName="revoke-keys" Oct 31 01:08:18.416665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.416656 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad935983-be86-4e2f-bf12-0aaa1bd8e716" containerName="revoke-keys" Oct 31 01:08:18.416767 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.416749 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad935983-be86-4e2f-bf12-0aaa1bd8e716" containerName="revoke-keys" Oct 31 01:08:18.416767 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.416763 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="61de143f-81b8-4cd1-b5b6-5f5dd780c071" containerName="payload-pre-processing" Oct 31 01:08:18.420421 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.420404 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.467102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.467073 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.467329 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.467300 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6m2p\" (UniqueName: \"kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.467496 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.467462 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.474408 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.474376 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:18.474513 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.474470 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:08:18.523465 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.523430 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:18.523465 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.523433 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:18.526320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.526295 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr"] Oct 31 01:08:18.569137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.569098 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6m2p\" (UniqueName: \"kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.569329 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.569272 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.569407 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.569377 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.570053 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.569973 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.571876 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.571853 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.637517 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.637449 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6m2p\" (UniqueName: \"kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p\") pod \"maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:18.734488 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:18.734417 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:19.119062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.119030 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr"] Oct 31 01:08:19.120363 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:08:19.120334 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cc6e3c_1281_49e9_811c_e34407010361.slice/crio-89c594e9f586015ab297e7238896ee60efd7c30cf97c8591ba08000181f6c612 WatchSource:0}: Error finding container 89c594e9f586015ab297e7238896ee60efd7c30cf97c8591ba08000181f6c612: Status 404 returned error can't find the container with id 89c594e9f586015ab297e7238896ee60efd7c30cf97c8591ba08000181f6c612 Oct 31 01:08:19.311110 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.311069 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" event={"ID":"d1cc6e3c-1281-49e9-811c-e34407010361","Type":"ContainerStarted","Data":"89c594e9f586015ab297e7238896ee60efd7c30cf97c8591ba08000181f6c612"} Oct 31 01:08:19.313023 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.312996 5943 generic.go:356] "Generic (PLEG): container finished" podID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerID="b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f" exitCode=0 Oct 31 01:08:19.313156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.313041 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" event={"ID":"ecf5c579-b3d6-429b-b91b-b45c109dfb09","Type":"ContainerDied","Data":"b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f"} Oct 31 01:08:19.313481 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.313467 5943 scope.go:122] "RemoveContainer" containerID="b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f" Oct 31 01:08:19.315072 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.315042 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:08:19.315158 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.315126 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:08:19.315212 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:19.315157 5943 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 01:08:20.321623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:20.321587 5943 generic.go:356] "Generic (PLEG): container finished" podID="d1cc6e3c-1281-49e9-811c-e34407010361" containerID="bc0a288945def0feabc9ac8424c5b166277e3f23d5969207ca908a66418f1a52" exitCode=0 Oct 31 01:08:20.321623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:20.321632 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" event={"ID":"d1cc6e3c-1281-49e9-811c-e34407010361","Type":"ContainerDied","Data":"bc0a288945def0feabc9ac8424c5b166277e3f23d5969207ca908a66418f1a52"} Oct 31 01:08:20.324138 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:20.324115 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" event={"ID":"ecf5c579-b3d6-429b-b91b-b45c109dfb09","Type":"ContainerStarted","Data":"cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec"} Oct 31 01:08:20.478857 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:20.478766 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:08:20.479034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:20.479013 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 01:08:20.479079 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:20.479062 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 01:08:21.478197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.478137 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 01:08:21.478603 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.478216 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 01:08:21.518396 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.518370 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:21.555006 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.554958 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:21.611503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.611417 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca\") pod \"d1cc6e3c-1281-49e9-811c-e34407010361\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " Oct 31 01:08:21.611503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.611472 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p\" (UniqueName: \"kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p\") pod \"d1cc6e3c-1281-49e9-811c-e34407010361\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " Oct 31 01:08:21.611751 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.611610 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca\") pod \"d1cc6e3c-1281-49e9-811c-e34407010361\" (UID: \"d1cc6e3c-1281-49e9-811c-e34407010361\") " Oct 31 01:08:21.611887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.611858 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca" pod "d1cc6e3c-1281-49e9-811c-e34407010361" (UID: "d1cc6e3c-1281-49e9-811c-e34407010361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:21.612148 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.612129 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:21.613763 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.613743 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca" pod "d1cc6e3c-1281-49e9-811c-e34407010361" (UID: "d1cc6e3c-1281-49e9-811c-e34407010361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:21.613974 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.613959 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p" pod "d1cc6e3c-1281-49e9-811c-e34407010361" (UID: "d1cc6e3c-1281-49e9-811c-e34407010361"). InnerVolumeSpecName "kube-api-access-k6m2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:21.700818 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.700782 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:21.712958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.712925 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cc6e3c-1281-49e9-811c-e34407010361-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:21.712958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:21.712954 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6m2p\" (UniqueName: \"kubernetes.io/projected/d1cc6e3c-1281-49e9-811c-e34407010361-kube-api-access-k6m2p\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:22.359877 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:22.359791 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" event={"ID":"d1cc6e3c-1281-49e9-811c-e34407010361","Type":"ContainerDied","Data":"89c594e9f586015ab297e7238896ee60efd7c30cf97c8591ba08000181f6c612"} Oct 31 01:08:22.359877 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:22.359821 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr" Oct 31 01:08:22.359877 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:22.359853 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c594e9f586015ab297e7238896ee60efd7c30cf97c8591ba08000181f6c612" Oct 31 01:08:22.478119 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:22.478075 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 01:08:22.478302 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:22.478137 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 01:08:22.915644 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:22.915597 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:23.064755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:23.064714 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:23.267648 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:23.267613 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:23.478950 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:23.478916 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 01:08:23.479389 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:23.478972 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 01:08:23.754980 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:23.754889 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:24.259540 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.259507 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws"] Oct 31 01:08:24.260099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.260079 5943 cpu_manager.go:414] "containerMap: removing container" podUID="d1cc6e3c-1281-49e9-811c-e34407010361" containerName="revoke-keys" Oct 31 01:08:24.260237 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.260101 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cc6e3c-1281-49e9-811c-e34407010361" containerName="revoke-keys" Oct 31 01:08:24.260373 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.260358 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cc6e3c-1281-49e9-811c-e34407010361" containerName="revoke-keys" Oct 31 01:08:24.262720 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.262701 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.289342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.289314 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fvw\" (UniqueName: \"kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.289488 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.289436 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.289488 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.289478 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.315293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.315261 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws"] Oct 31 01:08:24.362562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.362525 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:24.362562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.362553 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:24.362562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.362542 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:24.364066 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.364048 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:08:24.390806 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.390769 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fvw\" (UniqueName: \"kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.390975 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.390852 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.390975 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.390879 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.391568 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.391539 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.393424 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.393406 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.455887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.455855 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fvw\" (UniqueName: \"kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw\") pod \"maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.478743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.478712 5943 patch_prober.go:29] interesting pod/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 01:08:24.478906 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.478769 5943 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 01:08:24.576147 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.576044 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:24.983771 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:24.983747 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws"] Oct 31 01:08:24.985563 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:08:24.985539 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61531f8_a297_4f69_bacf_124265d1dba0.slice/crio-37e0f37ee94bff139a72604109a1d3d0647273f739060310e72269d9c5e44417 WatchSource:0}: Error finding container 37e0f37ee94bff139a72604109a1d3d0647273f739060310e72269d9c5e44417: Status 404 returned error can't find the container with id 37e0f37ee94bff139a72604109a1d3d0647273f739060310e72269d9c5e44417 Oct 31 01:08:25.418415 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:25.418381 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" event={"ID":"f61531f8-a297-4f69-bacf-124265d1dba0","Type":"ContainerStarted","Data":"dd807bcbea8e95d244ffdd6044d37a0f90f7d568e9fde11989afe90a012c0f59"} Oct 31 01:08:25.418415 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:25.418435 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" event={"ID":"f61531f8-a297-4f69-bacf-124265d1dba0","Type":"ContainerStarted","Data":"37e0f37ee94bff139a72604109a1d3d0647273f739060310e72269d9c5e44417"} Oct 31 01:08:25.533875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:25.533802 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" podStartSLOduration=1.5337865499999999 podStartE2EDuration="1.53378655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:25.532327509 +0000 UTC m=+3885.625617739" watchObservedRunningTime="2026-10-31 01:08:25.53378655 +0000 UTC m=+3885.627076736" Oct 31 01:08:25.540317 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:25.540292 5943 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:08:25.541300 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:25.541279 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:08:25.542327 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:25.542309 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:08:26.426468 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:26.426434 5943 generic.go:356] "Generic (PLEG): container finished" podID="f61531f8-a297-4f69-bacf-124265d1dba0" containerID="dd807bcbea8e95d244ffdd6044d37a0f90f7d568e9fde11989afe90a012c0f59" exitCode=0 Oct 31 01:08:26.426895 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:26.426521 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" event={"ID":"f61531f8-a297-4f69-bacf-124265d1dba0","Type":"ContainerDied","Data":"dd807bcbea8e95d244ffdd6044d37a0f90f7d568e9fde11989afe90a012c0f59"} Oct 31 01:08:26.946874 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:26.946832 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:27.480807 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:08:27.480764 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:08:27.592189 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.592150 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:27.628419 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.628392 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca\") pod \"f61531f8-a297-4f69-bacf-124265d1dba0\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " Oct 31 01:08:27.628608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.628455 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw\" (UniqueName: \"kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw\") pod \"f61531f8-a297-4f69-bacf-124265d1dba0\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " Oct 31 01:08:27.628608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.628548 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca\") pod \"f61531f8-a297-4f69-bacf-124265d1dba0\" (UID: \"f61531f8-a297-4f69-bacf-124265d1dba0\") " Oct 31 01:08:27.628911 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.628887 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca" pod "f61531f8-a297-4f69-bacf-124265d1dba0" (UID: "f61531f8-a297-4f69-bacf-124265d1dba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:27.629120 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.629100 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:27.630719 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.630694 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw" pod "f61531f8-a297-4f69-bacf-124265d1dba0" (UID: "f61531f8-a297-4f69-bacf-124265d1dba0"). InnerVolumeSpecName "kube-api-access-s4fvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:27.630820 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.630713 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca" pod "f61531f8-a297-4f69-bacf-124265d1dba0" (UID: "f61531f8-a297-4f69-bacf-124265d1dba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:27.730742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.730663 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61531f8-a297-4f69-bacf-124265d1dba0-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:27.730742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:27.730700 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4fvw\" (UniqueName: \"kubernetes.io/projected/f61531f8-a297-4f69-bacf-124265d1dba0-kube-api-access-s4fvw\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:28.441525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.441435 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" Oct 31 01:08:28.441525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.441463 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws" event={"ID":"f61531f8-a297-4f69-bacf-124265d1dba0","Type":"ContainerDied","Data":"37e0f37ee94bff139a72604109a1d3d0647273f739060310e72269d9c5e44417"} Oct 31 01:08:28.441525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.441518 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e0f37ee94bff139a72604109a1d3d0647273f739060310e72269d9c5e44417" Oct 31 01:08:28.714928 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.714825 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw"] Oct 31 01:08:28.727727 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.727692 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n"] Oct 31 01:08:28.736695 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.736663 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx"] Oct 31 01:08:28.885138 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:28.885104 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:29.494916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:29.494884 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:29.645004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:29.644964 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:31.959357 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:31.959316 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:32.922661 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:32.922622 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:36.108554 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:36.108517 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:36.256676 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:36.256613 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:36.840609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:36.840568 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:41.003437 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:41.003398 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:41.964320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:41.964287 5943 scope.go:122] "RemoveContainer" containerID="275a2e31ebf474b74f9c3e1505f5e5018e1dfb4bc8c87daf8729e8ee622c6194" Oct 31 01:08:42.509837 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:42.509797 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:42.761866 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:42.761770 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:42.906936 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:42.906899 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:43.013856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:43.013763 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg"] Oct 31 01:08:43.063126 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:43.063087 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-bf325c-jf7vg"] Oct 31 01:08:43.460048 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:43.459956 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:43.737135 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:43.737036 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" podUID="f4635629-dc5b-4a7b-b985-06319473adcf" containerName="main" containerID="cri-o://45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355" gracePeriod=45 Oct 31 01:08:43.750320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:43.750284 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" containerID="cri-o://77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9" gracePeriod=45 Oct 31 01:08:43.756903 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:43.756863 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" containerID="cri-o://8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b" gracePeriod=45 Oct 31 01:08:44.031661 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.031628 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 01:08:44.043625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.043588 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66a1b51-f158-47e5-81a8-a0f446d39a44" path="/var/lib/kubelet/pods/a66a1b51-f158-47e5-81a8-a0f446d39a44/volumes" Oct 31 01:08:44.076699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.076664 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 01:08:44.079998 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.079975 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 01:08:44.138452 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138413 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home\") pod \"f4635629-dc5b-4a7b-b985-06319473adcf\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " Oct 31 01:08:44.138671 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138542 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs\") pod \"f4635629-dc5b-4a7b-b985-06319473adcf\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " Oct 31 01:08:44.138671 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138602 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8\" (UniqueName: \"kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8\") pod \"f4635629-dc5b-4a7b-b985-06319473adcf\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " Oct 31 01:08:44.138819 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138775 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm\") pod \"f4635629-dc5b-4a7b-b985-06319473adcf\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " Oct 31 01:08:44.138819 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138797 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache\") pod \"f4635629-dc5b-4a7b-b985-06319473adcf\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " Oct 31 01:08:44.138939 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138834 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir\") pod \"f4635629-dc5b-4a7b-b985-06319473adcf\" (UID: \"f4635629-dc5b-4a7b-b985-06319473adcf\") " Oct 31 01:08:44.139379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.138820 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home" pod "f4635629-dc5b-4a7b-b985-06319473adcf" (UID: "f4635629-dc5b-4a7b-b985-06319473adcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.139379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.139220 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache" pod "f4635629-dc5b-4a7b-b985-06319473adcf" (UID: "f4635629-dc5b-4a7b-b985-06319473adcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.139379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.139254 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir" pod "f4635629-dc5b-4a7b-b985-06319473adcf" (UID: "f4635629-dc5b-4a7b-b985-06319473adcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.141244 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.141215 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm" pod "f4635629-dc5b-4a7b-b985-06319473adcf" (UID: "f4635629-dc5b-4a7b-b985-06319473adcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.141419 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.141400 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs" pod "f4635629-dc5b-4a7b-b985-06319473adcf" (UID: "f4635629-dc5b-4a7b-b985-06319473adcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:44.141573 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.141548 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8" pod "f4635629-dc5b-4a7b-b985-06319473adcf" (UID: "f4635629-dc5b-4a7b-b985-06319473adcf"). InnerVolumeSpecName "kube-api-access-x49x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.239668 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239640 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home\") pod \"2cca0779-c8e3-4252-86f7-d2ece966c43c\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " Oct 31 01:08:44.239867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239706 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir\") pod \"2cca0779-c8e3-4252-86f7-d2ece966c43c\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " Oct 31 01:08:44.239867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239734 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs\" (UniqueName: \"kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs\") pod \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " Oct 31 01:08:44.239867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239763 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache\") pod \"2cca0779-c8e3-4252-86f7-d2ece966c43c\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " Oct 31 01:08:44.239867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239804 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm\") pod \"2cca0779-c8e3-4252-86f7-d2ece966c43c\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " Oct 31 01:08:44.239867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239838 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg\" (UniqueName: \"kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg\") pod \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " Oct 31 01:08:44.240165 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239912 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home" pod "2cca0779-c8e3-4252-86f7-d2ece966c43c" (UID: "2cca0779-c8e3-4252-86f7-d2ece966c43c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.240165 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.239924 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache\") pod \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " Oct 31 01:08:44.240165 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240051 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs\") pod \"2cca0779-c8e3-4252-86f7-d2ece966c43c\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " Oct 31 01:08:44.240165 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240076 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache" pod "2cca0779-c8e3-4252-86f7-d2ece966c43c" (UID: "2cca0779-c8e3-4252-86f7-d2ece966c43c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.240165 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240116 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache" pod "84856bbd-a4ca-4463-9e0b-6f371f20ee61" (UID: "84856bbd-a4ca-4463-9e0b-6f371f20ee61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.240165 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240124 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home\") pod \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " Oct 31 01:08:44.240693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240198 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm\") pod \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " Oct 31 01:08:44.240693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240259 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir\") pod \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\" (UID: \"84856bbd-a4ca-4463-9e0b-6f371f20ee61\") " Oct 31 01:08:44.240693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240307 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir" pod "2cca0779-c8e3-4252-86f7-d2ece966c43c" (UID: "2cca0779-c8e3-4252-86f7-d2ece966c43c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.240693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240315 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k\" (UniqueName: \"kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k\") pod \"2cca0779-c8e3-4252-86f7-d2ece966c43c\" (UID: \"2cca0779-c8e3-4252-86f7-d2ece966c43c\") " Oct 31 01:08:44.240693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240464 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home" pod "84856bbd-a4ca-4463-9e0b-6f371f20ee61" (UID: "84856bbd-a4ca-4463-9e0b-6f371f20ee61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.240990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240892 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4635629-dc5b-4a7b-b985-06319473adcf-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.240990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240913 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x49x8\" (UniqueName: \"kubernetes.io/projected/f4635629-dc5b-4a7b-b985-06319473adcf-kube-api-access-x49x8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.240990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240928 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.240990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240939 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.240990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240952 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.240990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240986 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.241344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.241000 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.241344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.240995 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir" pod "84856bbd-a4ca-4463-9e0b-6f371f20ee61" (UID: "84856bbd-a4ca-4463-9e0b-6f371f20ee61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.241344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.241013 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.241344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.241026 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.241344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.241038 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.241344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.241050 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4635629-dc5b-4a7b-b985-06319473adcf-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.242534 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.242505 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs" pod "84856bbd-a4ca-4463-9e0b-6f371f20ee61" (UID: "84856bbd-a4ca-4463-9e0b-6f371f20ee61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:44.242703 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.242634 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm" pod "2cca0779-c8e3-4252-86f7-d2ece966c43c" (UID: "2cca0779-c8e3-4252-86f7-d2ece966c43c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.242854 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.242833 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs" pod "2cca0779-c8e3-4252-86f7-d2ece966c43c" (UID: "2cca0779-c8e3-4252-86f7-d2ece966c43c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:44.243156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.243123 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm" pod "84856bbd-a4ca-4463-9e0b-6f371f20ee61" (UID: "84856bbd-a4ca-4463-9e0b-6f371f20ee61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.243299 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.243277 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg" pod "84856bbd-a4ca-4463-9e0b-6f371f20ee61" (UID: "84856bbd-a4ca-4463-9e0b-6f371f20ee61"). InnerVolumeSpecName "kube-api-access-l8mdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.243477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.243461 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k" pod "2cca0779-c8e3-4252-86f7-d2ece966c43c" (UID: "2cca0779-c8e3-4252-86f7-d2ece966c43c"). InnerVolumeSpecName "kube-api-access-kl88k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.342522 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342487 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cca0779-c8e3-4252-86f7-d2ece966c43c-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.342522 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342525 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.342736 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342563 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.342736 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342581 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl88k\" (UniqueName: \"kubernetes.io/projected/2cca0779-c8e3-4252-86f7-d2ece966c43c-kube-api-access-kl88k\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.342736 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342598 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84856bbd-a4ca-4463-9e0b-6f371f20ee61-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.342736 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342612 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cca0779-c8e3-4252-86f7-d2ece966c43c-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.342736 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.342626 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8mdg\" (UniqueName: \"kubernetes.io/projected/84856bbd-a4ca-4463-9e0b-6f371f20ee61-kube-api-access-l8mdg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.572589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.572556 5943 generic.go:356] "Generic (PLEG): container finished" podID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerID="8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b" exitCode=0 Oct 31 01:08:44.572788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.572614 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" Oct 31 01:08:44.572788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.572632 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" event={"ID":"84856bbd-a4ca-4463-9e0b-6f371f20ee61","Type":"ContainerDied","Data":"8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b"} Oct 31 01:08:44.572788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.572671 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx" event={"ID":"84856bbd-a4ca-4463-9e0b-6f371f20ee61","Type":"ContainerDied","Data":"58c973a0a3a870d390e659d65717d76995c3aca4591986727231b72fa4cedad4"} Oct 31 01:08:44.572788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.572695 5943 scope.go:122] "RemoveContainer" containerID="8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b" Oct 31 01:08:44.574522 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.574493 5943 generic.go:356] "Generic (PLEG): container finished" podID="f4635629-dc5b-4a7b-b985-06319473adcf" containerID="45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355" exitCode=0 Oct 31 01:08:44.574669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.574548 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" Oct 31 01:08:44.574669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.574582 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" event={"ID":"f4635629-dc5b-4a7b-b985-06319473adcf","Type":"ContainerDied","Data":"45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355"} Oct 31 01:08:44.574669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.574624 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" event={"ID":"f4635629-dc5b-4a7b-b985-06319473adcf","Type":"ContainerDied","Data":"46fafdcf5a0759db13ff5dcabcd8f8b8b8133fb109517d1a6691b1cf6afad4cb"} Oct 31 01:08:44.576361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.576331 5943 generic.go:356] "Generic (PLEG): container finished" podID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerID="77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9" exitCode=0 Oct 31 01:08:44.576517 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.576422 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" Oct 31 01:08:44.576517 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.576434 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" event={"ID":"2cca0779-c8e3-4252-86f7-d2ece966c43c","Type":"ContainerDied","Data":"77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9"} Oct 31 01:08:44.576517 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.576470 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n" event={"ID":"2cca0779-c8e3-4252-86f7-d2ece966c43c","Type":"ContainerDied","Data":"bcc5c80465e53f39d42fe261c6e11d4f1a978f2134b6c8c778af1e69c53f2877"} Oct 31 01:08:44.584599 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.584582 5943 scope.go:122] "RemoveContainer" containerID="8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b" Oct 31 01:08:44.584896 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:08:44.584867 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b\": container with ID starting with 8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b not found: ID does not exist" containerID="8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b" Oct 31 01:08:44.584993 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.584909 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b"} err="failed to get container status \"8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b\": rpc error: code = NotFound desc = could not find container \"8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b\": container with ID starting with 8028e66e8a476b66d0c621fbd6514e9e0848f9d3e48a8e676604182de3f02c8b not found: ID does not exist" Oct 31 01:08:44.584993 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.584933 5943 scope.go:122] "RemoveContainer" containerID="45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355" Oct 31 01:08:44.594433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.594413 5943 scope.go:122] "RemoveContainer" containerID="45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355" Oct 31 01:08:44.594739 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:08:44.594717 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355\": container with ID starting with 45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355 not found: ID does not exist" containerID="45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355" Oct 31 01:08:44.594815 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.594751 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355"} err="failed to get container status \"45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355\": rpc error: code = NotFound desc = could not find container \"45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355\": container with ID starting with 45d65aff104c5abfe7e3a88f7903ad3c0ab61c831eeee6b5f110a4d221f1e355 not found: ID does not exist" Oct 31 01:08:44.594815 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.594777 5943 scope.go:122] "RemoveContainer" containerID="77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9" Oct 31 01:08:44.603729 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.603702 5943 scope.go:122] "RemoveContainer" containerID="77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9" Oct 31 01:08:44.603996 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:08:44.603977 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9\": container with ID starting with 77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9 not found: ID does not exist" containerID="77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9" Oct 31 01:08:44.604054 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.604006 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9"} err="failed to get container status \"77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9\": rpc error: code = NotFound desc = could not find container \"77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9\": container with ID starting with 77cd2e46affe1c4e52d416c351f67813d5183df3cb8f921758c2b31fe510b1c9 not found: ID does not exist" Oct 31 01:08:44.736765 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.736729 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n"] Oct 31 01:08:44.740883 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.740856 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/premium-w3-7abb95-kserve-f6847976f-qmd6n"] Oct 31 01:08:44.849711 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.849623 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx"] Oct 31 01:08:44.854229 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.854202 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/simulator-w3-7abb95-kserve-76cf7846d-rqhmx"] Oct 31 01:08:44.963632 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.963584 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw"] Oct 31 01:08:44.966017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:44.965995 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw"] Oct 31 01:08:45.002933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:45.002891 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-2-w3-7abb95-kserve-856866f5dc-774nw" podUID="f4635629-dc5b-4a7b-b985-06319473adcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:08:46.042193 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:46.042152 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" path="/var/lib/kubelet/pods/2cca0779-c8e3-4252-86f7-d2ece966c43c/volumes" Oct 31 01:08:46.042730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:46.042713 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" path="/var/lib/kubelet/pods/84856bbd-a4ca-4463-9e0b-6f371f20ee61/volumes" Oct 31 01:08:46.043114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:46.043101 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4635629-dc5b-4a7b-b985-06319473adcf" path="/var/lib/kubelet/pods/f4635629-dc5b-4a7b-b985-06319473adcf/volumes" Oct 31 01:08:47.971951 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:47.971921 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:49.729724 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:49.729680 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:49.890590 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:49.890547 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:50.115709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:50.115673 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:50.616605 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:50.616570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:51.109308 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109277 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl"] Oct 31 01:08:51.109790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109776 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f4635629-dc5b-4a7b-b985-06319473adcf" containerName="main" Oct 31 01:08:51.109845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109791 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4635629-dc5b-4a7b-b985-06319473adcf" containerName="main" Oct 31 01:08:51.109845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109808 5943 cpu_manager.go:414] "containerMap: removing container" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" Oct 31 01:08:51.109845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109813 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" Oct 31 01:08:51.109845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109833 5943 cpu_manager.go:414] "containerMap: removing container" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" Oct 31 01:08:51.109845 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109838 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" Oct 31 01:08:51.109996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109853 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f61531f8-a297-4f69-bacf-124265d1dba0" containerName="revoke-keys" Oct 31 01:08:51.109996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109858 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61531f8-a297-4f69-bacf-124265d1dba0" containerName="revoke-keys" Oct 31 01:08:51.109996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109961 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4635629-dc5b-4a7b-b985-06319473adcf" containerName="main" Oct 31 01:08:51.109996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109968 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="84856bbd-a4ca-4463-9e0b-6f371f20ee61" containerName="main" Oct 31 01:08:51.109996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109974 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cca0779-c8e3-4252-86f7-d2ece966c43c" containerName="main" Oct 31 01:08:51.109996 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.109982 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61531f8-a297-4f69-bacf-124265d1dba0" containerName="revoke-keys" Oct 31 01:08:51.115419 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.115398 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.121735 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.121711 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.121871 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.121751 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlrd\" (UniqueName: \"kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.121981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.121964 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.172682 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.172644 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl"] Oct 31 01:08:51.219126 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.219088 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:51.219126 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.219097 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:51.219483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.219156 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:51.219483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.219222 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:08:51.222913 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.222886 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.223049 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.222991 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.223049 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.223039 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlrd\" (UniqueName: \"kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.223539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.223519 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.225489 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.225467 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.284128 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.284095 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlrd\" (UniqueName: \"kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd\") pod \"maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.428268 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.428163 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:51.806772 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:51.806745 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl"] Oct 31 01:08:51.809527 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:08:51.809495 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6fb808_e46e_43ad_86a9_33798ed2f5cf.slice/crio-582d44b2eee7f7225aecb7a6d07feae93b7bbf9fa47af5edee302c85a0e13356 WatchSource:0}: Error finding container 582d44b2eee7f7225aecb7a6d07feae93b7bbf9fa47af5edee302c85a0e13356: Status 404 returned error can't find the container with id 582d44b2eee7f7225aecb7a6d07feae93b7bbf9fa47af5edee302c85a0e13356 Oct 31 01:08:52.636089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:52.636053 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" event={"ID":"3f6fb808-e46e-43ad-86a9-33798ed2f5cf","Type":"ContainerStarted","Data":"218bac3806717ea248ae67971bcb89d480534375e3da40cddcc856a3ccc04aed"} Oct 31 01:08:52.636538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:52.636122 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" event={"ID":"3f6fb808-e46e-43ad-86a9-33798ed2f5cf","Type":"ContainerStarted","Data":"582d44b2eee7f7225aecb7a6d07feae93b7bbf9fa47af5edee302c85a0e13356"} Oct 31 01:08:52.750745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:52.750696 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" podStartSLOduration=1.7506773 podStartE2EDuration="1.7506773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:52.750483344 +0000 UTC m=+3912.843773552" watchObservedRunningTime="2026-10-31 01:08:52.7506773 +0000 UTC m=+3912.843967486" Oct 31 01:08:53.228382 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:53.228344 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:53.643804 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:53.643768 5943 generic.go:356] "Generic (PLEG): container finished" podID="3f6fb808-e46e-43ad-86a9-33798ed2f5cf" containerID="218bac3806717ea248ae67971bcb89d480534375e3da40cddcc856a3ccc04aed" exitCode=0 Oct 31 01:08:53.644255 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:53.643853 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" event={"ID":"3f6fb808-e46e-43ad-86a9-33798ed2f5cf","Type":"ContainerDied","Data":"218bac3806717ea248ae67971bcb89d480534375e3da40cddcc856a3ccc04aed"} Oct 31 01:08:54.792637 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.792610 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:54.860777 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.860746 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca\") pod \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " Oct 31 01:08:54.860961 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.860824 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca\") pod \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " Oct 31 01:08:54.860961 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.860923 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd\" (UniqueName: \"kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd\") pod \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\" (UID: \"3f6fb808-e46e-43ad-86a9-33798ed2f5cf\") " Oct 31 01:08:54.861218 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.861158 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca" pod "3f6fb808-e46e-43ad-86a9-33798ed2f5cf" (UID: "3f6fb808-e46e-43ad-86a9-33798ed2f5cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:54.861482 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.861461 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:54.863130 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.863100 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca" pod "3f6fb808-e46e-43ad-86a9-33798ed2f5cf" (UID: "3f6fb808-e46e-43ad-86a9-33798ed2f5cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:54.863232 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.863209 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd" pod "3f6fb808-e46e-43ad-86a9-33798ed2f5cf" (UID: "3f6fb808-e46e-43ad-86a9-33798ed2f5cf"). InnerVolumeSpecName "kube-api-access-2zlrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:54.963084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.962996 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:54.963084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:54.963030 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlrd\" (UniqueName: \"kubernetes.io/projected/3f6fb808-e46e-43ad-86a9-33798ed2f5cf-kube-api-access-2zlrd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:55.121867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:55.121818 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:08:55.580677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:55.580641 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w"] Oct 31 01:08:55.659107 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:55.659070 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" Oct 31 01:08:55.659275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:55.659103 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl" event={"ID":"3f6fb808-e46e-43ad-86a9-33798ed2f5cf","Type":"ContainerDied","Data":"582d44b2eee7f7225aecb7a6d07feae93b7bbf9fa47af5edee302c85a0e13356"} Oct 31 01:08:55.659275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:55.659155 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="582d44b2eee7f7225aecb7a6d07feae93b7bbf9fa47af5edee302c85a0e13356" Oct 31 01:08:55.704537 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:55.704501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:57.477041 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:08:57.477000 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:08:58.477886 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:58.477851 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:59.427293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:08:59.427237 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:00.785330 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:00.785296 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:00.929813 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:00.929775 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:01.614984 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:01.614943 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:02.038883 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:02.038846 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:02.782554 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:02.782520 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:02.979867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:02.979833 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:03.126101 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:03.125986 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:04.820495 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:04.820441 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.021401 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:05.021358 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.163963 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:05.163874 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.382403 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:05.382366 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx"] Oct 31 01:09:05.389102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:05.389072 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e8808d-njcpx"] Oct 31 01:09:05.813117 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:05.813084 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.982797 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:05.982749 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.041071 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:06.041034 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ee1761-9e3b-4cf3-b6de-c0b134f62e57" path="/var/lib/kubelet/pods/18ee1761-9e3b-4cf3-b6de-c0b134f62e57/volumes" Oct 31 01:09:06.152625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:06.152526 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.651693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:06.651657 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.798544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:06.798505 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:07.423544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:07.423509 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.495506 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.495470 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.602561 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.602518 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" containerID="cri-o://56d474696e033f6250356c7f3e86c4ae8b9c3c2fec3e24edc207b40c8c253101" gracePeriod=45 Oct 31 01:09:10.765305 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.765274 5943 generic.go:356] "Generic (PLEG): container finished" podID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerID="56d474696e033f6250356c7f3e86c4ae8b9c3c2fec3e24edc207b40c8c253101" exitCode=0 Oct 31 01:09:10.765480 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.765343 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" event={"ID":"67acf536-3524-471c-babf-7a1af4cc4a4e","Type":"ContainerDied","Data":"56d474696e033f6250356c7f3e86c4ae8b9c3c2fec3e24edc207b40c8c253101"} Oct 31 01:09:10.864129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.864068 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 01:09:10.966020 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.965988 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs\") pod \"67acf536-3524-471c-babf-7a1af4cc4a4e\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " Oct 31 01:09:10.966219 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966049 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home\") pod \"67acf536-3524-471c-babf-7a1af4cc4a4e\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " Oct 31 01:09:10.966219 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966073 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd\" (UniqueName: \"kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd\") pod \"67acf536-3524-471c-babf-7a1af4cc4a4e\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " Oct 31 01:09:10.966219 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966156 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir\") pod \"67acf536-3524-471c-babf-7a1af4cc4a4e\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " Oct 31 01:09:10.966390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966236 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache\") pod \"67acf536-3524-471c-babf-7a1af4cc4a4e\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " Oct 31 01:09:10.966390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966276 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm\") pod \"67acf536-3524-471c-babf-7a1af4cc4a4e\" (UID: \"67acf536-3524-471c-babf-7a1af4cc4a4e\") " Oct 31 01:09:10.966390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966351 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home" pod "67acf536-3524-471c-babf-7a1af4cc4a4e" (UID: "67acf536-3524-471c-babf-7a1af4cc4a4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:10.966547 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966523 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir" pod "67acf536-3524-471c-babf-7a1af4cc4a4e" (UID: "67acf536-3524-471c-babf-7a1af4cc4a4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:10.966599 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966556 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache" pod "67acf536-3524-471c-babf-7a1af4cc4a4e" (UID: "67acf536-3524-471c-babf-7a1af4cc4a4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:10.966773 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966752 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:10.966773 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966775 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:10.966948 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.966789 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:10.968544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.968519 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm" pod "67acf536-3524-471c-babf-7a1af4cc4a4e" (UID: "67acf536-3524-471c-babf-7a1af4cc4a4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:10.968635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.968552 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs" pod "67acf536-3524-471c-babf-7a1af4cc4a4e" (UID: "67acf536-3524-471c-babf-7a1af4cc4a4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:10.968677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.968637 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd" pod "67acf536-3524-471c-babf-7a1af4cc4a4e" (UID: "67acf536-3524-471c-babf-7a1af4cc4a4e"). InnerVolumeSpecName "kube-api-access-h6whd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:10.976504 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:10.976479 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.068620 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.068540 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67acf536-3524-471c-babf-7a1af4cc4a4e-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.068620 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.068569 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67acf536-3524-471c-babf-7a1af4cc4a4e-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.068620 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.068580 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6whd\" (UniqueName: \"kubernetes.io/projected/67acf536-3524-471c-babf-7a1af4cc4a4e-kube-api-access-h6whd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.154915 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.154864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.773363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.773330 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" event={"ID":"67acf536-3524-471c-babf-7a1af4cc4a4e","Type":"ContainerDied","Data":"61be546ab7ba44cbf15cb8ff0a28b5805f3435c4b357387a4163e6dff20e4aa5"} Oct 31 01:09:11.773363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.773367 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w" Oct 31 01:09:11.773904 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.773380 5943 scope.go:122] "RemoveContainer" containerID="56d474696e033f6250356c7f3e86c4ae8b9c3c2fec3e24edc207b40c8c253101" Oct 31 01:09:11.936576 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.936537 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w"] Oct 31 01:09:11.985479 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:11.985449 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-w2-e1ca71-kserve-647d5f889-4z26w"] Oct 31 01:09:12.040727 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:12.040647 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" path="/var/lib/kubelet/pods/67acf536-3524-471c-babf-7a1af4cc4a4e/volumes" Oct 31 01:09:12.663791 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:12.663757 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:12.812654 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:12.812616 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.056477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:13.056444 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4"] Oct 31 01:09:13.056682 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:13.056660 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" containerID="cri-o://f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762" gracePeriod=30 Oct 31 01:09:13.059570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:13.059537 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.547118 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:13.547077 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.739629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:13.739585 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.131803 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.131762 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx"] Oct 31 01:09:14.132576 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.132555 5943 cpu_manager.go:414] "containerMap: removing container" podUID="3f6fb808-e46e-43ad-86a9-33798ed2f5cf" containerName="revoke-keys" Oct 31 01:09:14.132643 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.132580 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6fb808-e46e-43ad-86a9-33798ed2f5cf" containerName="revoke-keys" Oct 31 01:09:14.132643 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.132597 5943 cpu_manager.go:414] "containerMap: removing container" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" Oct 31 01:09:14.132643 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.132613 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" Oct 31 01:09:14.132760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.132736 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6fb808-e46e-43ad-86a9-33798ed2f5cf" containerName="revoke-keys" Oct 31 01:09:14.132760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.132745 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="67acf536-3524-471c-babf-7a1af4cc4a4e" containerName="main" Oct 31 01:09:14.136131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.136103 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.188562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.188525 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:09:14.238288 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.238252 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:09:14.238513 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.238481 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:09:14.240864 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.240839 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx"] Oct 31 01:09:14.288742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.288710 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:09:14.326746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.326714 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.326746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.326752 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.326991 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.326973 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgb2\" (UniqueName: \"kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.428355 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.428272 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgb2\" (UniqueName: \"kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.428355 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.428333 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.428355 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.428355 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.429073 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.429043 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.431039 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.431016 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.500457 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.500424 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgb2\" (UniqueName: \"kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2\") pod \"maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:14.750413 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:14.750317 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:15.130392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:15.130367 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx"] Oct 31 01:09:15.132543 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:09:15.132514 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51643a7c_c4bc_4851_9dad_4880d094c2dc.slice/crio-51473f9db230eb85fa40aeca56d3afe7e477fa930a2e08707882f7cedfccaf0c WatchSource:0}: Error finding container 51473f9db230eb85fa40aeca56d3afe7e477fa930a2e08707882f7cedfccaf0c: Status 404 returned error can't find the container with id 51473f9db230eb85fa40aeca56d3afe7e477fa930a2e08707882f7cedfccaf0c Oct 31 01:09:15.697471 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:15.697383 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:15.803356 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:15.803319 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" event={"ID":"51643a7c-c4bc-4851-9dad-4880d094c2dc","Type":"ContainerStarted","Data":"aaa626ce1b8f2cf9bebaeded8f9b0e5252ce13c8a67e7fa146dca50daf31d152"} Oct 31 01:09:15.803537 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:15.803374 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" event={"ID":"51643a7c-c4bc-4851-9dad-4880d094c2dc","Type":"ContainerStarted","Data":"51473f9db230eb85fa40aeca56d3afe7e477fa930a2e08707882f7cedfccaf0c"} Oct 31 01:09:15.920637 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:15.920575 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" podStartSLOduration=1.920560177 podStartE2EDuration="1.920560177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:15.917897661 +0000 UTC m=+3936.011187851" watchObservedRunningTime="2026-10-31 01:09:15.920560177 +0000 UTC m=+3936.013850363" Oct 31 01:09:16.812089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:16.812054 5943 generic.go:356] "Generic (PLEG): container finished" podID="51643a7c-c4bc-4851-9dad-4880d094c2dc" containerID="aaa626ce1b8f2cf9bebaeded8f9b0e5252ce13c8a67e7fa146dca50daf31d152" exitCode=0 Oct 31 01:09:16.812567 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:16.812139 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" event={"ID":"51643a7c-c4bc-4851-9dad-4880d094c2dc","Type":"ContainerDied","Data":"aaa626ce1b8f2cf9bebaeded8f9b0e5252ce13c8a67e7fa146dca50daf31d152"} Oct 31 01:09:17.978442 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:17.978408 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:18.068949 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.068870 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2\" (UniqueName: \"kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2\") pod \"51643a7c-c4bc-4851-9dad-4880d094c2dc\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " Oct 31 01:09:18.069140 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.069047 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca\") pod \"51643a7c-c4bc-4851-9dad-4880d094c2dc\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " Oct 31 01:09:18.069140 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.069073 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca\") pod \"51643a7c-c4bc-4851-9dad-4880d094c2dc\" (UID: \"51643a7c-c4bc-4851-9dad-4880d094c2dc\") " Oct 31 01:09:18.069449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.069420 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca" pod "51643a7c-c4bc-4851-9dad-4880d094c2dc" (UID: "51643a7c-c4bc-4851-9dad-4880d094c2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:18.071310 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.071279 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2" pod "51643a7c-c4bc-4851-9dad-4880d094c2dc" (UID: "51643a7c-c4bc-4851-9dad-4880d094c2dc"). InnerVolumeSpecName "kube-api-access-txgb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:18.071485 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.071462 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca" pod "51643a7c-c4bc-4851-9dad-4880d094c2dc" (UID: "51643a7c-c4bc-4851-9dad-4880d094c2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:18.170152 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.170130 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txgb2\" (UniqueName: \"kubernetes.io/projected/51643a7c-c4bc-4851-9dad-4880d094c2dc-kube-api-access-txgb2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.170241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.170159 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.170241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.170171 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51643a7c-c4bc-4851-9dad-4880d094c2dc-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.366913 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.366887 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:09:18.472671 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.472638 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.472873 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.472691 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.472873 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.472791 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.472873 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.472834 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.473029 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.472916 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.473029 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.472996 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.473145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473030 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.473145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473107 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:18.473145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473138 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.473359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473243 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv\") pod \"b46b4d69-1555-473c-916d-7d466222de4a\" (UID: \"b46b4d69-1555-473c-916d-7d466222de4a\") " Oct 31 01:09:18.473359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473313 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:18.473484 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473395 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:18.473484 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473416 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:18.473801 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473768 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:18.473945 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473857 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.473945 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473882 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46b4d69-1555-473c-916d-7d466222de4a-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.473945 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473897 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.473945 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.473911 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.475556 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.475534 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "kube-api-access-j66gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:18.475705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.475684 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:18.475810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.475787 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:09:18.475861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.475817 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy" pod "b46b4d69-1555-473c-916d-7d466222de4a" (UID: "b46b4d69-1555-473c-916d-7d466222de4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:18.561423 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.561383 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r"] Oct 31 01:09:18.568529 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.568498 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d"] Oct 31 01:09:18.575680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.575611 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.576260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.576232 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46b4d69-1555-473c-916d-7d466222de4a-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.576375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.576263 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j66gv\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-kube-api-access-j66gv\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.576375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.576279 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46b4d69-1555-473c-916d-7d466222de4a-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.576375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.576292 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46b4d69-1555-473c-916d-7d466222de4a-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.578637 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.578616 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b"] Oct 31 01:09:18.684803 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.684715 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:18.826587 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.826556 5943 generic.go:356] "Generic (PLEG): container finished" podID="b46b4d69-1555-473c-916d-7d466222de4a" containerID="f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762" exitCode=0 Oct 31 01:09:18.826788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.826617 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" Oct 31 01:09:18.826788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.826640 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" event={"ID":"b46b4d69-1555-473c-916d-7d466222de4a","Type":"ContainerDied","Data":"f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762"} Oct 31 01:09:18.826788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.826738 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4" event={"ID":"b46b4d69-1555-473c-916d-7d466222de4a","Type":"ContainerDied","Data":"c515425b9bcb3a4108b1632631732ee96c56ba1096f7f48555accc5973f5a46d"} Oct 31 01:09:18.826788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.826766 5943 scope.go:122] "RemoveContainer" containerID="f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762" Oct 31 01:09:18.828489 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.828456 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" Oct 31 01:09:18.828489 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.828481 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx" event={"ID":"51643a7c-c4bc-4851-9dad-4880d094c2dc","Type":"ContainerDied","Data":"51473f9db230eb85fa40aeca56d3afe7e477fa930a2e08707882f7cedfccaf0c"} Oct 31 01:09:18.828684 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.828526 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51473f9db230eb85fa40aeca56d3afe7e477fa930a2e08707882f7cedfccaf0c" Oct 31 01:09:18.838330 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.838308 5943 scope.go:122] "RemoveContainer" containerID="c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b" Oct 31 01:09:18.847728 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.847708 5943 scope.go:122] "RemoveContainer" containerID="f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762" Oct 31 01:09:18.848036 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:18.848017 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762\": container with ID starting with f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762 not found: ID does not exist" containerID="f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762" Oct 31 01:09:18.848116 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.848049 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762"} err="failed to get container status \"f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762\": rpc error: code = NotFound desc = could not find container \"f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762\": container with ID starting with f463d9962075bcf664a8d892d9612cad93b828319bf3ee131b5a6fea6b9be762 not found: ID does not exist" Oct 31 01:09:18.848116 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.848075 5943 scope.go:122] "RemoveContainer" containerID="c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b" Oct 31 01:09:18.848364 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:18.848344 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b\": container with ID starting with c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b not found: ID does not exist" containerID="c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b" Oct 31 01:09:18.848414 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:18.848371 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b"} err="failed to get container status \"c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b\": rpc error: code = NotFound desc = could not find container \"c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b\": container with ID starting with c7148095c06f5ed5b330913e65f6b59bf4f5df5dad607ccd801320bf8fd5c84b not found: ID does not exist" Oct 31 01:09:19.045764 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:19.045729 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4"] Oct 31 01:09:19.048991 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:19.048964 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e8808d-openshift-default-bdd8b8b7c-6lmp4"] Oct 31 01:09:20.040694 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:20.040658 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46b4d69-1555-473c-916d-7d466222de4a" path="/var/lib/kubelet/pods/b46b4d69-1555-473c-916d-7d466222de4a/volumes" Oct 31 01:09:20.157078 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:20.157039 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:21.319814 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.319778 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd"] Oct 31 01:09:21.320297 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320282 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" Oct 31 01:09:21.320363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320299 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" Oct 31 01:09:21.320363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320314 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" Oct 31 01:09:21.320363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320320 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" Oct 31 01:09:21.320363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320327 5943 cpu_manager.go:414] "containerMap: removing container" podUID="51643a7c-c4bc-4851-9dad-4880d094c2dc" containerName="revoke-keys" Oct 31 01:09:21.320363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320332 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51643a7c-c4bc-4851-9dad-4880d094c2dc" containerName="revoke-keys" Oct 31 01:09:21.320549 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320431 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" Oct 31 01:09:21.320549 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.320440 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="51643a7c-c4bc-4851-9dad-4880d094c2dc" containerName="revoke-keys" Oct 31 01:09:21.325161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.325144 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.379289 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.379255 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb"] Oct 31 01:09:21.379998 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.379952 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46b4d69-1555-473c-916d-7d466222de4a" containerName="istio-proxy" Oct 31 01:09:21.383453 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.383432 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.418444 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.418411 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78fvd\" (UniqueName: \"kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.418644 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.418457 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.418644 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.418592 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.418644 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.418634 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.418846 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.418670 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgd66\" (UniqueName: \"kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.418846 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.418695 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.426344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.426316 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:09:21.426344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.426334 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:09:21.426344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.426351 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:09:21.426344 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.426334 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:09:21.431760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.431736 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd"] Oct 31 01:09:21.448353 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.448328 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb"] Oct 31 01:09:21.520228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.520150 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78fvd\" (UniqueName: \"kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.520460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.520250 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.520460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.520355 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.520460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.520422 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.520647 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.520484 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgd66\" (UniqueName: \"kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.520647 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.520530 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.521140 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.521114 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.521281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.521214 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.522819 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.522799 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.522916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.522880 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.581711 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.581640 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78fvd\" (UniqueName: \"kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd\") pod \"maas-api-revoke-sub-e4740656-7hkkb\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.581857 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.581821 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgd66\" (UniqueName: \"kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66\") pod \"maas-api-revoke-sub-5afc23b4-mlcwd\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.639131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.639101 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:21.699657 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.699633 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:21.715876 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.715849 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:21.993875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:21.993777 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:22.017236 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.017209 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd"] Oct 31 01:09:22.019224 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:09:22.019190 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f70a13_1039_4ee3_9bb3_26ff9dea2af0.slice/crio-eb2e1898ce5c7689706a394838a87486bd31df45eb824622f681e91cfbfe3f4a WatchSource:0}: Error finding container eb2e1898ce5c7689706a394838a87486bd31df45eb824622f681e91cfbfe3f4a: Status 404 returned error can't find the container with id eb2e1898ce5c7689706a394838a87486bd31df45eb824622f681e91cfbfe3f4a Oct 31 01:09:22.118839 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.118808 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb"] Oct 31 01:09:22.120710 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:09:22.120687 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75d3154_1a7b_437e_99dc_92f967abab44.slice/crio-92e65762f5f0affbb29b0d462fde9028204d366155f40ef17ef9b4977b89db3f WatchSource:0}: Error finding container 92e65762f5f0affbb29b0d462fde9028204d366155f40ef17ef9b4977b89db3f: Status 404 returned error can't find the container with id 92e65762f5f0affbb29b0d462fde9028204d366155f40ef17ef9b4977b89db3f Oct 31 01:09:22.155351 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.155318 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:22.214779 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.214708 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr"] Oct 31 01:09:22.217489 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.217461 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-395b06-gvczr"] Oct 31 01:09:22.701947 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.701912 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:22.856944 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.856908 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" event={"ID":"60f70a13-1039-4ee3-9bb3-26ff9dea2af0","Type":"ContainerStarted","Data":"94378f3aa13ae1937eebebcdc47d5fbfb854fec4218a07945de8ff285b477aaa"} Oct 31 01:09:22.857148 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.856998 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" event={"ID":"60f70a13-1039-4ee3-9bb3-26ff9dea2af0","Type":"ContainerStarted","Data":"eb2e1898ce5c7689706a394838a87486bd31df45eb824622f681e91cfbfe3f4a"} Oct 31 01:09:22.858704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.858681 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" event={"ID":"c75d3154-1a7b-437e-99dc-92f967abab44","Type":"ContainerStarted","Data":"8390ec50bbf9b3650334b34e43e87dbde8437ee5595018ffe23ac6042722d506"} Oct 31 01:09:22.858811 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.858719 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" event={"ID":"c75d3154-1a7b-437e-99dc-92f967abab44","Type":"ContainerStarted","Data":"92e65762f5f0affbb29b0d462fde9028204d366155f40ef17ef9b4977b89db3f"} Oct 31 01:09:22.974143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:22.974040 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" podStartSLOduration=1.9740251340000001 podStartE2EDuration="1.974025134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:22.972063145 +0000 UTC m=+3943.065353373" watchObservedRunningTime="2026-10-31 01:09:22.974025134 +0000 UTC m=+3943.067315320" Oct 31 01:09:23.090139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:23.090078 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" podStartSLOduration=2.090062599 podStartE2EDuration="2.090062599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:23.08787733 +0000 UTC m=+3943.181167517" watchObservedRunningTime="2026-10-31 01:09:23.090062599 +0000 UTC m=+3943.183352788" Oct 31 01:09:23.867385 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:23.867339 5943 generic.go:356] "Generic (PLEG): container finished" podID="60f70a13-1039-4ee3-9bb3-26ff9dea2af0" containerID="94378f3aa13ae1937eebebcdc47d5fbfb854fec4218a07945de8ff285b477aaa" exitCode=0 Oct 31 01:09:23.867830 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:23.867387 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" event={"ID":"60f70a13-1039-4ee3-9bb3-26ff9dea2af0","Type":"ContainerDied","Data":"94378f3aa13ae1937eebebcdc47d5fbfb854fec4218a07945de8ff285b477aaa"} Oct 31 01:09:23.869198 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:23.869159 5943 generic.go:356] "Generic (PLEG): container finished" podID="c75d3154-1a7b-437e-99dc-92f967abab44" containerID="8390ec50bbf9b3650334b34e43e87dbde8437ee5595018ffe23ac6042722d506" exitCode=0 Oct 31 01:09:23.869313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:23.869208 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" event={"ID":"c75d3154-1a7b-437e-99dc-92f967abab44","Type":"ContainerDied","Data":"8390ec50bbf9b3650334b34e43e87dbde8437ee5595018ffe23ac6042722d506"} Oct 31 01:09:24.045417 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:24.043661 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cc6e3c-1281-49e9-811c-e34407010361" path="/var/lib/kubelet/pods/d1cc6e3c-1281-49e9-811c-e34407010361/volumes" Oct 31 01:09:25.043753 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.043721 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:25.073728 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.073702 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:25.166461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166383 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd\" (UniqueName: \"kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd\") pod \"c75d3154-1a7b-437e-99dc-92f967abab44\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " Oct 31 01:09:25.166461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166421 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66\" (UniqueName: \"kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66\") pod \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " Oct 31 01:09:25.166714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166474 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca\") pod \"c75d3154-1a7b-437e-99dc-92f967abab44\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " Oct 31 01:09:25.166714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166494 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca\") pod \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " Oct 31 01:09:25.166714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166582 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca\") pod \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\" (UID: \"60f70a13-1039-4ee3-9bb3-26ff9dea2af0\") " Oct 31 01:09:25.166714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166641 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca\") pod \"c75d3154-1a7b-437e-99dc-92f967abab44\" (UID: \"c75d3154-1a7b-437e-99dc-92f967abab44\") " Oct 31 01:09:25.166965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.166931 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca" pod "60f70a13-1039-4ee3-9bb3-26ff9dea2af0" (UID: "60f70a13-1039-4ee3-9bb3-26ff9dea2af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:25.167760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.167221 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca" pod "c75d3154-1a7b-437e-99dc-92f967abab44" (UID: "c75d3154-1a7b-437e-99dc-92f967abab44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:25.167760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.167435 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:25.167760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.167456 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:25.169228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.169076 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca" pod "c75d3154-1a7b-437e-99dc-92f967abab44" (UID: "c75d3154-1a7b-437e-99dc-92f967abab44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:25.169329 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.169312 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66" pod "60f70a13-1039-4ee3-9bb3-26ff9dea2af0" (UID: "60f70a13-1039-4ee3-9bb3-26ff9dea2af0"). InnerVolumeSpecName "kube-api-access-xgd66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:25.169414 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.169384 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca" pod "60f70a13-1039-4ee3-9bb3-26ff9dea2af0" (UID: "60f70a13-1039-4ee3-9bb3-26ff9dea2af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:25.169621 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.169598 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd" pod "c75d3154-1a7b-437e-99dc-92f967abab44" (UID: "c75d3154-1a7b-437e-99dc-92f967abab44"). InnerVolumeSpecName "kube-api-access-78fvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:25.268347 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.268306 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78fvd\" (UniqueName: \"kubernetes.io/projected/c75d3154-1a7b-437e-99dc-92f967abab44-kube-api-access-78fvd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:25.268347 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.268343 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgd66\" (UniqueName: \"kubernetes.io/projected/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-kube-api-access-xgd66\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:25.268347 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.268354 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75d3154-1a7b-437e-99dc-92f967abab44-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:25.268623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.268364 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f70a13-1039-4ee3-9bb3-26ff9dea2af0-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:25.885541 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.885512 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" Oct 31 01:09:25.885710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.885640 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb" event={"ID":"c75d3154-1a7b-437e-99dc-92f967abab44","Type":"ContainerDied","Data":"92e65762f5f0affbb29b0d462fde9028204d366155f40ef17ef9b4977b89db3f"} Oct 31 01:09:25.885710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.885689 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92e65762f5f0affbb29b0d462fde9028204d366155f40ef17ef9b4977b89db3f" Oct 31 01:09:25.887241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.887221 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" event={"ID":"60f70a13-1039-4ee3-9bb3-26ff9dea2af0","Type":"ContainerDied","Data":"eb2e1898ce5c7689706a394838a87486bd31df45eb824622f681e91cfbfe3f4a"} Oct 31 01:09:25.887359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.887252 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2e1898ce5c7689706a394838a87486bd31df45eb824622f681e91cfbfe3f4a" Oct 31 01:09:25.887359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:25.887300 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd" Oct 31 01:09:26.075911 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:26.075821 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:26.267439 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:26.267403 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:27.241889 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:27.241850 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:27.489283 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:27.489242 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:09:29.037574 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:29.037530 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:29.190161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:29.190116 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:29.398067 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:29.397969 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l"] Oct 31 01:09:29.398259 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:29.398199 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" containerID="cri-o://e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843" gracePeriod=30 Oct 31 01:09:29.409867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:29.409834 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:30.413755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.413719 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k"] Oct 31 01:09:30.414293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.414276 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c75d3154-1a7b-437e-99dc-92f967abab44" containerName="revoke-keys" Oct 31 01:09:30.414358 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.414296 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75d3154-1a7b-437e-99dc-92f967abab44" containerName="revoke-keys" Oct 31 01:09:30.414358 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.414322 5943 cpu_manager.go:414] "containerMap: removing container" podUID="60f70a13-1039-4ee3-9bb3-26ff9dea2af0" containerName="revoke-keys" Oct 31 01:09:30.414358 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.414327 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f70a13-1039-4ee3-9bb3-26ff9dea2af0" containerName="revoke-keys" Oct 31 01:09:30.414459 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.414441 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75d3154-1a7b-437e-99dc-92f967abab44" containerName="revoke-keys" Oct 31 01:09:30.414496 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.414458 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f70a13-1039-4ee3-9bb3-26ff9dea2af0" containerName="revoke-keys" Oct 31 01:09:30.417546 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.417528 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.469982 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.469942 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:09:30.519400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.519356 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:09:30.519400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.519377 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:09:30.519624 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.519480 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:09:30.521917 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.521894 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k"] Oct 31 01:09:30.528156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.528120 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5h66\" (UniqueName: \"kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.528273 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.528162 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.528340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.528316 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.630105 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.630072 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5h66\" (UniqueName: \"kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.630353 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.630117 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.630353 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.630191 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.630838 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.630810 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.632859 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.632836 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.696120 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.696047 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5h66\" (UniqueName: \"kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:30.731307 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:30.731269 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:31.110476 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.110453 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k"] Oct 31 01:09:31.112615 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:09:31.112580 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0af2a45_8b92_4b39_8da1_a1c946ba9efa.slice/crio-926dbb457a77e3fcf36b19b3fa65052767e7a768603b338a965d14962a2b424c WatchSource:0}: Error finding container 926dbb457a77e3fcf36b19b3fa65052767e7a768603b338a965d14962a2b424c: Status 404 returned error can't find the container with id 926dbb457a77e3fcf36b19b3fa65052767e7a768603b338a965d14962a2b424c Oct 31 01:09:31.591823 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.591786 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr"] Oct 31 01:09:31.595731 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.595714 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.651346 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.651311 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp"] Oct 31 01:09:31.655638 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.655613 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr"] Oct 31 01:09:31.655855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.655734 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.722350 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.722263 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp"] Oct 31 01:09:31.742581 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.742542 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.742581 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.742584 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.742867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.742664 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jw7\" (UniqueName: \"kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.742867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.742788 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhpf\" (UniqueName: \"kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.742995 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.742904 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.743059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.743011 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.844024 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.843986 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.844024 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.844030 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.844312 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.844173 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jw7\" (UniqueName: \"kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.844312 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.844277 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhpf\" (UniqueName: \"kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.844423 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.844373 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.844541 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.844484 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.844859 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.844834 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.845343 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.845322 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.846817 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.846798 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.847068 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.847050 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.905066 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.905035 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jw7\" (UniqueName: \"kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7\") pod \"maas-api-revoke-sub-2769324f-9mbtr\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.905066 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.905062 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhpf\" (UniqueName: \"kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf\") pod \"maas-api-revoke-sub-8e682311-bd7sp\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:31.909897 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.909874 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:31.926897 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.926863 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" event={"ID":"c0af2a45-8b92-4b39-8da1-a1c946ba9efa","Type":"ContainerStarted","Data":"6957e625888ce551f8b07019d8c513843f7f167c923b66658f26ddc0319fe2db"} Oct 31 01:09:31.927019 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.926957 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" event={"ID":"c0af2a45-8b92-4b39-8da1-a1c946ba9efa","Type":"ContainerStarted","Data":"926dbb457a77e3fcf36b19b3fa65052767e7a768603b338a965d14962a2b424c"} Oct 31 01:09:31.980546 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:31.980513 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:32.050486 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.050433 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" podStartSLOduration=2.050417983 podStartE2EDuration="2.050417983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:32.047981238 +0000 UTC m=+3952.141271435" watchObservedRunningTime="2026-10-31 01:09:32.050417983 +0000 UTC m=+3952.143708191" Oct 31 01:09:32.293964 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.293936 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr"] Oct 31 01:09:32.295107 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:09:32.295078 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f725819_eab9_4b01_b31f_84f2812a9428.slice/crio-5fa6def52e15ee1c9affb3dbef50fe2896dae233f10bfe938301e307a909ad60 WatchSource:0}: Error finding container 5fa6def52e15ee1c9affb3dbef50fe2896dae233f10bfe938301e307a909ad60: Status 404 returned error can't find the container with id 5fa6def52e15ee1c9affb3dbef50fe2896dae233f10bfe938301e307a909ad60 Oct 31 01:09:32.404371 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.404332 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp"] Oct 31 01:09:32.406345 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:09:32.406314 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9820980_ef20_441e_8a6e_fed2c8229744.slice/crio-7cbfc32f956744afaa62201b5d818e9ff1858608b63846afa45cafaa3cd19124 WatchSource:0}: Error finding container 7cbfc32f956744afaa62201b5d818e9ff1858608b63846afa45cafaa3cd19124: Status 404 returned error can't find the container with id 7cbfc32f956744afaa62201b5d818e9ff1858608b63846afa45cafaa3cd19124 Oct 31 01:09:32.488311 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.488278 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:32.937429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.937342 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" event={"ID":"b9820980-ef20-441e-8a6e-fed2c8229744","Type":"ContainerStarted","Data":"2bc656a5660e7af6af1965f5f02c35c52544cd800a95ff64e0b280b696c8edfd"} Oct 31 01:09:32.937429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.937402 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" event={"ID":"b9820980-ef20-441e-8a6e-fed2c8229744","Type":"ContainerStarted","Data":"7cbfc32f956744afaa62201b5d818e9ff1858608b63846afa45cafaa3cd19124"} Oct 31 01:09:32.939629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.939578 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" event={"ID":"3f725819-eab9-4b01-b31f-84f2812a9428","Type":"ContainerStarted","Data":"49f8d7a903bdbf8780592e068a2be55fb6953698f8205bfa1178b05aa070f889"} Oct 31 01:09:32.939986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.939649 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" event={"ID":"3f725819-eab9-4b01-b31f-84f2812a9428","Type":"ContainerStarted","Data":"5fa6def52e15ee1c9affb3dbef50fe2896dae233f10bfe938301e307a909ad60"} Oct 31 01:09:32.941903 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.941870 5943 generic.go:356] "Generic (PLEG): container finished" podID="c0af2a45-8b92-4b39-8da1-a1c946ba9efa" containerID="6957e625888ce551f8b07019d8c513843f7f167c923b66658f26ddc0319fe2db" exitCode=0 Oct 31 01:09:32.942348 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:32.941966 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" event={"ID":"c0af2a45-8b92-4b39-8da1-a1c946ba9efa","Type":"ContainerDied","Data":"6957e625888ce551f8b07019d8c513843f7f167c923b66658f26ddc0319fe2db"} Oct 31 01:09:33.057309 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.057239 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" podStartSLOduration=2.057220334 podStartE2EDuration="2.057220334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:33.055426559 +0000 UTC m=+3953.148716771" watchObservedRunningTime="2026-10-31 01:09:33.057220334 +0000 UTC m=+3953.150510523" Oct 31 01:09:33.288778 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.288717 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" podStartSLOduration=2.288698537 podStartE2EDuration="2.288698537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:33.285196155 +0000 UTC m=+3953.378486340" watchObservedRunningTime="2026-10-31 01:09:33.288698537 +0000 UTC m=+3953.381988727" Oct 31 01:09:33.583779 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.583681 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" containerID="cri-o://3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9" gracePeriod=45 Oct 31 01:09:33.594720 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.594684 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" containerID="cri-o://5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee" gracePeriod=45 Oct 31 01:09:33.599633 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.599594 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" containerID="cri-o://cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c" gracePeriod=45 Oct 31 01:09:33.613282 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.613049 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Oct 31 01:09:33.678076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.677981 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:33.878904 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.878851 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:33.926074 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.925696 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 01:09:33.947235 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.947205 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 01:09:33.952805 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.952773 5943 generic.go:356] "Generic (PLEG): container finished" podID="b9820980-ef20-441e-8a6e-fed2c8229744" containerID="2bc656a5660e7af6af1965f5f02c35c52544cd800a95ff64e0b280b696c8edfd" exitCode=0 Oct 31 01:09:33.952960 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.952856 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" event={"ID":"b9820980-ef20-441e-8a6e-fed2c8229744","Type":"ContainerDied","Data":"2bc656a5660e7af6af1965f5f02c35c52544cd800a95ff64e0b280b696c8edfd"} Oct 31 01:09:33.953289 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.953266 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 01:09:33.954588 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.954565 5943 generic.go:356] "Generic (PLEG): container finished" podID="3f725819-eab9-4b01-b31f-84f2812a9428" containerID="49f8d7a903bdbf8780592e068a2be55fb6953698f8205bfa1178b05aa070f889" exitCode=0 Oct 31 01:09:33.954700 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.954646 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" event={"ID":"3f725819-eab9-4b01-b31f-84f2812a9428","Type":"ContainerDied","Data":"49f8d7a903bdbf8780592e068a2be55fb6953698f8205bfa1178b05aa070f889"} Oct 31 01:09:33.956215 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.956169 5943 generic.go:356] "Generic (PLEG): container finished" podID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerID="5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee" exitCode=0 Oct 31 01:09:33.956318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.956232 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" Oct 31 01:09:33.956318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.956296 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" event={"ID":"50c74fcf-a456-45ae-a09d-4cbbe59714ba","Type":"ContainerDied","Data":"5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee"} Oct 31 01:09:33.956449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.956334 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d" event={"ID":"50c74fcf-a456-45ae-a09d-4cbbe59714ba","Type":"ContainerDied","Data":"acaf1553671790d794b9cda75e86eb7d201890a8eeea296e59d161eb85d2deed"} Oct 31 01:09:33.956449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.956358 5943 scope.go:122] "RemoveContainer" containerID="5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee" Oct 31 01:09:33.957714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.957692 5943 generic.go:356] "Generic (PLEG): container finished" podID="b781bdd4-3456-4da2-aa92-021f64272877" containerID="cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c" exitCode=0 Oct 31 01:09:33.957836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.957743 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" Oct 31 01:09:33.957836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.957760 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" event={"ID":"b781bdd4-3456-4da2-aa92-021f64272877","Type":"ContainerDied","Data":"cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c"} Oct 31 01:09:33.957836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.957786 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b" event={"ID":"b781bdd4-3456-4da2-aa92-021f64272877","Type":"ContainerDied","Data":"9c60e6127e92655573598e520a38c0b231fc7c5a0b065ec43562aa963f0a5349"} Oct 31 01:09:33.959438 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.959413 5943 generic.go:356] "Generic (PLEG): container finished" podID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerID="3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9" exitCode=0 Oct 31 01:09:33.959539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.959472 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" Oct 31 01:09:33.959539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.959493 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" event={"ID":"a67438e1-87b4-4676-ba1b-964c3e3bcedb","Type":"ContainerDied","Data":"3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9"} Oct 31 01:09:33.959539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.959528 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r" event={"ID":"a67438e1-87b4-4676-ba1b-964c3e3bcedb","Type":"ContainerDied","Data":"9ca5859f7e0fbef75dd48ed450385b89fa68b3ae40807054892dab9b2cc846e9"} Oct 31 01:09:33.969304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.969280 5943 scope.go:122] "RemoveContainer" containerID="5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee" Oct 31 01:09:33.969645 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:33.969623 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee\": container with ID starting with 5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee not found: ID does not exist" containerID="5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee" Oct 31 01:09:33.969739 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.969654 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee"} err="failed to get container status \"5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee\": rpc error: code = NotFound desc = could not find container \"5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee\": container with ID starting with 5a553da14b580f72253fdcba59dcebd927e7d9191ab8703a4d4fcf6c2a5814ee not found: ID does not exist" Oct 31 01:09:33.969739 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.969670 5943 scope.go:122] "RemoveContainer" containerID="cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c" Oct 31 01:09:33.980923 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.980905 5943 scope.go:122] "RemoveContainer" containerID="cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c" Oct 31 01:09:33.981310 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:33.981286 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c\": container with ID starting with cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c not found: ID does not exist" containerID="cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c" Oct 31 01:09:33.981405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.981315 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c"} err="failed to get container status \"cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c\": rpc error: code = NotFound desc = could not find container \"cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c\": container with ID starting with cbe9506a4da77d96a54ab4155578ef9a98c631dfb6bcdc9746847482f23a846c not found: ID does not exist" Oct 31 01:09:33.981405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.981334 5943 scope.go:122] "RemoveContainer" containerID="3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9" Oct 31 01:09:33.993563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.993541 5943 scope.go:122] "RemoveContainer" containerID="3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9" Oct 31 01:09:33.994402 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:33.994316 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9\": container with ID starting with 3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9 not found: ID does not exist" containerID="3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9" Oct 31 01:09:33.994402 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:33.994359 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9"} err="failed to get container status \"3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9\": rpc error: code = NotFound desc = could not find container \"3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9\": container with ID starting with 3943976d4666381b7914e4bb751962bf3a7d7e0d0067922defb7c3b6d2a020e9 not found: ID does not exist" Oct 31 01:09:34.031198 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.031135 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:34.079462 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079425 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs\") pod \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " Oct 31 01:09:34.079690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079490 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home\") pod \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " Oct 31 01:09:34.079690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079522 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm\") pod \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " Oct 31 01:09:34.079690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079583 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs\" (UniqueName: \"kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs\") pod \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " Oct 31 01:09:34.079690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079614 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir\") pod \"b781bdd4-3456-4da2-aa92-021f64272877\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " Oct 31 01:09:34.079690 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079649 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs\") pod \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079711 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home\") pod \"b781bdd4-3456-4da2-aa92-021f64272877\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079759 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir\") pod \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079803 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache\") pod \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079833 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm\") pod \"b781bdd4-3456-4da2-aa92-021f64272877\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079866 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm\") pod \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079896 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache\") pod \"b781bdd4-3456-4da2-aa92-021f64272877\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079926 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir\") pod \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " Oct 31 01:09:34.079979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.079956 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs\" (UniqueName: \"kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs\") pod \"b781bdd4-3456-4da2-aa92-021f64272877\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " Oct 31 01:09:34.080485 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.080035 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6\" (UniqueName: \"kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6\") pod \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " Oct 31 01:09:34.080485 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.080077 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks\" (UniqueName: \"kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks\") pod \"b781bdd4-3456-4da2-aa92-021f64272877\" (UID: \"b781bdd4-3456-4da2-aa92-021f64272877\") " Oct 31 01:09:34.080485 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.080210 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home\") pod \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\" (UID: \"a67438e1-87b4-4676-ba1b-964c3e3bcedb\") " Oct 31 01:09:34.080485 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.080240 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache\") pod \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\" (UID: \"50c74fcf-a456-45ae-a09d-4cbbe59714ba\") " Oct 31 01:09:34.082022 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.081701 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir" pod "b781bdd4-3456-4da2-aa92-021f64272877" (UID: "b781bdd4-3456-4da2-aa92-021f64272877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.082131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.082034 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home" pod "b781bdd4-3456-4da2-aa92-021f64272877" (UID: "b781bdd4-3456-4da2-aa92-021f64272877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.082395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.082292 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache" pod "50c74fcf-a456-45ae-a09d-4cbbe59714ba" (UID: "50c74fcf-a456-45ae-a09d-4cbbe59714ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.082706 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.082516 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir" pod "50c74fcf-a456-45ae-a09d-4cbbe59714ba" (UID: "50c74fcf-a456-45ae-a09d-4cbbe59714ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.083077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.082816 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home" pod "50c74fcf-a456-45ae-a09d-4cbbe59714ba" (UID: "50c74fcf-a456-45ae-a09d-4cbbe59714ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.083464 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.083444 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache" pod "a67438e1-87b4-4676-ba1b-964c3e3bcedb" (UID: "a67438e1-87b4-4676-ba1b-964c3e3bcedb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.083754 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.083736 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache" pod "b781bdd4-3456-4da2-aa92-021f64272877" (UID: "b781bdd4-3456-4da2-aa92-021f64272877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.084257 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.084231 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs" pod "a67438e1-87b4-4676-ba1b-964c3e3bcedb" (UID: "a67438e1-87b4-4676-ba1b-964c3e3bcedb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:34.084490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.084466 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir" pod "a67438e1-87b4-4676-ba1b-964c3e3bcedb" (UID: "a67438e1-87b4-4676-ba1b-964c3e3bcedb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.085565 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.085267 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs" pod "b781bdd4-3456-4da2-aa92-021f64272877" (UID: "b781bdd4-3456-4da2-aa92-021f64272877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:34.085655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.085587 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home" pod "a67438e1-87b4-4676-ba1b-964c3e3bcedb" (UID: "a67438e1-87b4-4676-ba1b-964c3e3bcedb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.087027 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.086966 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm" pod "50c74fcf-a456-45ae-a09d-4cbbe59714ba" (UID: "50c74fcf-a456-45ae-a09d-4cbbe59714ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.087531 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.087510 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm" pod "a67438e1-87b4-4676-ba1b-964c3e3bcedb" (UID: "a67438e1-87b4-4676-ba1b-964c3e3bcedb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.091544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.090473 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs" pod "a67438e1-87b4-4676-ba1b-964c3e3bcedb" (UID: "a67438e1-87b4-4676-ba1b-964c3e3bcedb"). InnerVolumeSpecName "kube-api-access-wk2zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:34.091544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.090543 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs" pod "50c74fcf-a456-45ae-a09d-4cbbe59714ba" (UID: "50c74fcf-a456-45ae-a09d-4cbbe59714ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:34.091544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.091069 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6" pod "50c74fcf-a456-45ae-a09d-4cbbe59714ba" (UID: "50c74fcf-a456-45ae-a09d-4cbbe59714ba"). InnerVolumeSpecName "kube-api-access-tbfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:34.091852 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.091828 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks" pod "b781bdd4-3456-4da2-aa92-021f64272877" (UID: "b781bdd4-3456-4da2-aa92-021f64272877"). InnerVolumeSpecName "kube-api-access-g8gks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:34.091925 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.091473 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm" pod "b781bdd4-3456-4da2-aa92-021f64272877" (UID: "b781bdd4-3456-4da2-aa92-021f64272877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.114318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.114288 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:34.181591 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181552 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181591 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181589 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181591 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181599 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181607 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181638 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181647 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781bdd4-3456-4da2-aa92-021f64272877-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181657 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfb6\" (UniqueName: \"kubernetes.io/projected/50c74fcf-a456-45ae-a09d-4cbbe59714ba-kube-api-access-tbfb6\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181667 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8gks\" (UniqueName: \"kubernetes.io/projected/b781bdd4-3456-4da2-aa92-021f64272877-kube-api-access-g8gks\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181676 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181684 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181692 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67438e1-87b4-4676-ba1b-964c3e3bcedb-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181699 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181706 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67438e1-87b4-4676-ba1b-964c3e3bcedb-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181714 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2zs\" (UniqueName: \"kubernetes.io/projected/a67438e1-87b4-4676-ba1b-964c3e3bcedb-kube-api-access-wk2zs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181721 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181729 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181737 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781bdd4-3456-4da2-aa92-021f64272877-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.181858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.181744 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c74fcf-a456-45ae-a09d-4cbbe59714ba-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.282123 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.282083 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca\") pod \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " Oct 31 01:09:34.282304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.282284 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca\") pod \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " Oct 31 01:09:34.282379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.282347 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66\" (UniqueName: \"kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66\") pod \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\" (UID: \"c0af2a45-8b92-4b39-8da1-a1c946ba9efa\") " Oct 31 01:09:34.282507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.282481 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca" pod "c0af2a45-8b92-4b39-8da1-a1c946ba9efa" (UID: "c0af2a45-8b92-4b39-8da1-a1c946ba9efa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:34.282723 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.282700 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.284629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.284608 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca" pod "c0af2a45-8b92-4b39-8da1-a1c946ba9efa" (UID: "c0af2a45-8b92-4b39-8da1-a1c946ba9efa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:34.284714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.284648 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66" pod "c0af2a45-8b92-4b39-8da1-a1c946ba9efa" (UID: "c0af2a45-8b92-4b39-8da1-a1c946ba9efa"). InnerVolumeSpecName "kube-api-access-b5h66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:34.383948 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.383871 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.383948 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.383902 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5h66\" (UniqueName: \"kubernetes.io/projected/c0af2a45-8b92-4b39-8da1-a1c946ba9efa-kube-api-access-b5h66\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.501672 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.501643 5943 patch_prober.go:29] interesting pod/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:09:34.501833 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.501694 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:09:34.651939 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.651914 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:09:34.680466 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.680429 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r"] Oct 31 01:09:34.684559 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.684527 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-w1-869cca-kserve-54cdd69cf4-jff7r"] Oct 31 01:09:34.788198 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788150 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788417 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788325 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788417 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788391 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788417 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788442 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788462 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788479 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788523 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788630 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs\") pod \"16b824e8-840c-4564-980a-80a4fdb5c205\" (UID: \"16b824e8-840c-4564-980a-80a4fdb5c205\") " Oct 31 01:09:34.788810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788669 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.788810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788759 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.788810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.788785 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:34.789094 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789072 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.789159 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789102 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.789355 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789334 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.789429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789361 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b824e8-840c-4564-980a-80a4fdb5c205-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.789429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789377 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.789429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789391 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.789429 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.789405 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.790699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.790672 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:34.790820 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.790786 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "kube-api-access-knhbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:34.791349 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.791334 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:34.791452 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.791432 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo" pod "16b824e8-840c-4564-980a-80a4fdb5c205" (UID: "16b824e8-840c-4564-980a-80a4fdb5c205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:09:34.890046 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.890012 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knhbg\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-kube-api-access-knhbg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.890046 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.890042 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b824e8-840c-4564-980a-80a4fdb5c205-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.890046 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.890054 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b824e8-840c-4564-980a-80a4fdb5c205-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.890336 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.890063 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b824e8-840c-4564-980a-80a4fdb5c205-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:34.909018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.908940 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b"] Oct 31 01:09:34.913212 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.913167 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/simulator-w1-869cca-kserve-7ff4785566-gb24b"] Oct 31 01:09:34.969677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.969649 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" Oct 31 01:09:34.969677 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.969655 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k" event={"ID":"c0af2a45-8b92-4b39-8da1-a1c946ba9efa","Type":"ContainerDied","Data":"926dbb457a77e3fcf36b19b3fa65052767e7a768603b338a965d14962a2b424c"} Oct 31 01:09:34.970225 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.969744 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="926dbb457a77e3fcf36b19b3fa65052767e7a768603b338a965d14962a2b424c" Oct 31 01:09:34.973873 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.973846 5943 generic.go:356] "Generic (PLEG): container finished" podID="16b824e8-840c-4564-980a-80a4fdb5c205" containerID="e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843" exitCode=0 Oct 31 01:09:34.973979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.973942 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" Oct 31 01:09:34.974017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.973974 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" event={"ID":"16b824e8-840c-4564-980a-80a4fdb5c205","Type":"ContainerDied","Data":"e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843"} Oct 31 01:09:34.974051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.974026 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l" event={"ID":"16b824e8-840c-4564-980a-80a4fdb5c205","Type":"ContainerDied","Data":"2b7df1947e78178e5ed66259e78206ab2ce6c31b7263396159e7b80322d6a35c"} Oct 31 01:09:34.974096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.974050 5943 scope.go:122] "RemoveContainer" containerID="e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843" Oct 31 01:09:34.991132 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:34.991101 5943 scope.go:122] "RemoveContainer" containerID="be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6" Oct 31 01:09:35.002803 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.002768 5943 scope.go:122] "RemoveContainer" containerID="e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843" Oct 31 01:09:35.003306 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:35.003166 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843\": container with ID starting with e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843 not found: ID does not exist" containerID="e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843" Oct 31 01:09:35.003428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.003320 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843"} err="failed to get container status \"e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843\": rpc error: code = NotFound desc = could not find container \"e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843\": container with ID starting with e4c8c7d1e69bc7753d3055f9837741fa67a240d41b786e847d49f79cbec9d843 not found: ID does not exist" Oct 31 01:09:35.003428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.003340 5943 scope.go:122] "RemoveContainer" containerID="be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6" Oct 31 01:09:35.003669 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:35.003647 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6\": container with ID starting with be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6 not found: ID does not exist" containerID="be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6" Oct 31 01:09:35.003757 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.003679 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6"} err="failed to get container status \"be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6\": rpc error: code = NotFound desc = could not find container \"be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6\": container with ID starting with be019eb02fa3886e0c509cf0224456a5bf536ea70a18e92f870647f260f4a8a6 not found: ID does not exist" Oct 31 01:09:35.024470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.024369 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d"] Oct 31 01:09:35.029225 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.028487 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/premium-w1-869cca-kserve-f49c566dc-jg47d"] Oct 31 01:09:35.151516 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.151486 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:35.153496 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.153480 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:35.261081 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.261049 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l"] Oct 31 01:09:35.265888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.265863 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-395b06-openshift-default-6644689b57-xqt4l"] Oct 31 01:09:35.294244 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294217 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf\" (UniqueName: \"kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf\") pod \"b9820980-ef20-441e-8a6e-fed2c8229744\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " Oct 31 01:09:35.294433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294309 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca\") pod \"3f725819-eab9-4b01-b31f-84f2812a9428\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " Oct 31 01:09:35.294433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294347 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca\") pod \"3f725819-eab9-4b01-b31f-84f2812a9428\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " Oct 31 01:09:35.294433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294374 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca\") pod \"b9820980-ef20-441e-8a6e-fed2c8229744\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " Oct 31 01:09:35.294433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294392 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca\") pod \"b9820980-ef20-441e-8a6e-fed2c8229744\" (UID: \"b9820980-ef20-441e-8a6e-fed2c8229744\") " Oct 31 01:09:35.294660 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294468 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7\" (UniqueName: \"kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7\") pod \"3f725819-eab9-4b01-b31f-84f2812a9428\" (UID: \"3f725819-eab9-4b01-b31f-84f2812a9428\") " Oct 31 01:09:35.294780 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294757 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca" pod "3f725819-eab9-4b01-b31f-84f2812a9428" (UID: "3f725819-eab9-4b01-b31f-84f2812a9428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:35.294851 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294797 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca" pod "b9820980-ef20-441e-8a6e-fed2c8229744" (UID: "b9820980-ef20-441e-8a6e-fed2c8229744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:35.294942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294927 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:35.295010 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.294947 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:35.296574 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.296553 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf" pod "b9820980-ef20-441e-8a6e-fed2c8229744" (UID: "b9820980-ef20-441e-8a6e-fed2c8229744"). InnerVolumeSpecName "kube-api-access-ghhpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:35.296679 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.296654 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7" pod "3f725819-eab9-4b01-b31f-84f2812a9428" (UID: "3f725819-eab9-4b01-b31f-84f2812a9428"). InnerVolumeSpecName "kube-api-access-p6jw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:35.297054 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.297028 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca" pod "b9820980-ef20-441e-8a6e-fed2c8229744" (UID: "b9820980-ef20-441e-8a6e-fed2c8229744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:35.297054 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.297033 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca" pod "3f725819-eab9-4b01-b31f-84f2812a9428" (UID: "3f725819-eab9-4b01-b31f-84f2812a9428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:35.396405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.396371 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jw7\" (UniqueName: \"kubernetes.io/projected/3f725819-eab9-4b01-b31f-84f2812a9428-kube-api-access-p6jw7\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:35.396405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.396398 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhpf\" (UniqueName: \"kubernetes.io/projected/b9820980-ef20-441e-8a6e-fed2c8229744-kube-api-access-ghhpf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:35.396405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.396410 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f725819-eab9-4b01-b31f-84f2812a9428-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:35.396658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.396418 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9820980-ef20-441e-8a6e-fed2c8229744-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:35.983085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.982981 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" event={"ID":"b9820980-ef20-441e-8a6e-fed2c8229744","Type":"ContainerDied","Data":"7cbfc32f956744afaa62201b5d818e9ff1858608b63846afa45cafaa3cd19124"} Oct 31 01:09:35.983085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.983038 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cbfc32f956744afaa62201b5d818e9ff1858608b63846afa45cafaa3cd19124" Oct 31 01:09:35.983085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.982997 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp" Oct 31 01:09:35.984666 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.984643 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" Oct 31 01:09:35.984801 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.984676 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr" event={"ID":"3f725819-eab9-4b01-b31f-84f2812a9428","Type":"ContainerDied","Data":"5fa6def52e15ee1c9affb3dbef50fe2896dae233f10bfe938301e307a909ad60"} Oct 31 01:09:35.984801 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:35.984715 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa6def52e15ee1c9affb3dbef50fe2896dae233f10bfe938301e307a909ad60" Oct 31 01:09:36.041395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:36.041365 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" path="/var/lib/kubelet/pods/16b824e8-840c-4564-980a-80a4fdb5c205/volumes" Oct 31 01:09:36.041855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:36.041841 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" path="/var/lib/kubelet/pods/50c74fcf-a456-45ae-a09d-4cbbe59714ba/volumes" Oct 31 01:09:36.042270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:36.042257 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" path="/var/lib/kubelet/pods/a67438e1-87b4-4676-ba1b-964c3e3bcedb/volumes" Oct 31 01:09:36.042952 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:36.042937 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b781bdd4-3456-4da2-aa92-021f64272877" path="/var/lib/kubelet/pods/b781bdd4-3456-4da2-aa92-021f64272877/volumes" Oct 31 01:09:37.537836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:37.537788 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:39.760929 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:39.760892 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.342779 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:42.342693 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.488417 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:42.488372 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.750528 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:45.750491 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:09:45.979372 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:45.979341 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:47.235327 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:47.235291 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:51.844839 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:51.844739 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:52.030212 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:52.030161 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:52.176121 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:52.176013 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:56.451272 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:56.451149 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:56.653332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:56.653290 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:56.802459 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:56.802424 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:56.952706 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:09:56.952663 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:57.477592 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:09:57.477546 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:10:01.130934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.130889 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:01.156172 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156131 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh"] Oct 31 01:10:01.156646 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156633 5943 cpu_manager.go:414] "containerMap: removing container" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156648 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156658 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c0af2a45-8b92-4b39-8da1-a1c946ba9efa" containerName="revoke-keys" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156663 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0af2a45-8b92-4b39-8da1-a1c946ba9efa" containerName="revoke-keys" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156676 5943 cpu_manager.go:414] "containerMap: removing container" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156681 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156697 5943 cpu_manager.go:414] "containerMap: removing container" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" Oct 31 01:10:01.156705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156702 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156710 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b9820980-ef20-441e-8a6e-fed2c8229744" containerName="revoke-keys" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156715 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9820980-ef20-441e-8a6e-fed2c8229744" containerName="revoke-keys" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156725 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156729 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156743 5943 cpu_manager.go:414] "containerMap: removing container" podUID="3f725819-eab9-4b01-b31f-84f2812a9428" containerName="revoke-keys" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156747 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f725819-eab9-4b01-b31f-84f2812a9428" containerName="revoke-keys" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156816 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67438e1-87b4-4676-ba1b-964c3e3bcedb" containerName="main" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156823 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0af2a45-8b92-4b39-8da1-a1c946ba9efa" containerName="revoke-keys" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156832 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156839 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c74fcf-a456-45ae-a09d-4cbbe59714ba" containerName="main" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156845 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9820980-ef20-441e-8a6e-fed2c8229744" containerName="revoke-keys" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156851 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b781bdd4-3456-4da2-aa92-021f64272877" containerName="main" Oct 31 01:10:01.157015 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.156857 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f725819-eab9-4b01-b31f-84f2812a9428" containerName="revoke-keys" Oct 31 01:10:01.161379 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.161357 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.204957 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.204922 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r"] Oct 31 01:10:01.205445 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.205432 5943 cpu_manager.go:414] "containerMap: removing container" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" Oct 31 01:10:01.205506 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.205447 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" Oct 31 01:10:01.205562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.205552 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b824e8-840c-4564-980a-80a4fdb5c205" containerName="istio-proxy" Oct 31 01:10:01.208940 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.208925 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.252245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.252210 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:10:01.252245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.252226 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:10:01.252496 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.252252 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:10:01.254709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.254683 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh"] Oct 31 01:10:01.265857 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.265829 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:10:01.268792 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.268764 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r"] Oct 31 01:10:01.305381 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.305341 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spxbb\" (UniqueName: \"kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.305597 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.305457 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.305597 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.305498 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.406685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.406593 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spxbb\" (UniqueName: \"kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.406685 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.406673 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.406933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.406707 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.406933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.406735 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.406933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.406776 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplks\" (UniqueName: \"kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.407072 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.406954 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.407412 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.407387 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.409264 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.409244 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.467663 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.467630 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spxbb\" (UniqueName: \"kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb\") pod \"maas-api-revoke-sub-5b5bdfdb-sbmwh\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.475503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.475475 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:01.508519 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.508468 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.508725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.508533 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplks\" (UniqueName: \"kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.508725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.508577 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.509257 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.509232 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.511477 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.511455 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.570459 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.570430 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplks\" (UniqueName: \"kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks\") pod \"maas-api-revoke-sub-3e40865f-7r78r\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.824820 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.824782 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:01.855348 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.855313 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh"] Oct 31 01:10:01.856040 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:10:01.856012 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bb5cda_962d_4b64_855e_29bc388a375e.slice/crio-20584d3812880f49c19cf15e81f9d7d483df6c56a9fb73d6a93d89e9c964c980 WatchSource:0}: Error finding container 20584d3812880f49c19cf15e81f9d7d483df6c56a9fb73d6a93d89e9c964c980: Status 404 returned error can't find the container with id 20584d3812880f49c19cf15e81f9d7d483df6c56a9fb73d6a93d89e9c964c980 Oct 31 01:10:01.959172 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:01.959137 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:02.156127 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:02.156087 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" event={"ID":"82bb5cda-962d-4b64-855e-29bc388a375e","Type":"ContainerStarted","Data":"0d708b6846d3a97ecedff9a25a5b45355259e34ea599eb686651f54468288628"} Oct 31 01:10:02.156662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:02.156152 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" event={"ID":"82bb5cda-962d-4b64-855e-29bc388a375e","Type":"ContainerStarted","Data":"20584d3812880f49c19cf15e81f9d7d483df6c56a9fb73d6a93d89e9c964c980"} Oct 31 01:10:02.205488 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:02.205456 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r"] Oct 31 01:10:02.208213 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:10:02.208167 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2eed7b_85fa_4762_a7a5_3024f050d52e.slice/crio-1b686d0ff9047be212178fc6baa32fe84b2d8a85fcf2b5330f8176539e0c0fc3 WatchSource:0}: Error finding container 1b686d0ff9047be212178fc6baa32fe84b2d8a85fcf2b5330f8176539e0c0fc3: Status 404 returned error can't find the container with id 1b686d0ff9047be212178fc6baa32fe84b2d8a85fcf2b5330f8176539e0c0fc3 Oct 31 01:10:02.272410 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:02.272347 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" podStartSLOduration=1.272331523 podStartE2EDuration="1.272331523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:02.270971209 +0000 UTC m=+3982.364261415" watchObservedRunningTime="2026-10-31 01:10:02.272331523 +0000 UTC m=+3982.365621710" Oct 31 01:10:02.708033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:02.707259 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:03.165157 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:03.165122 5943 generic.go:356] "Generic (PLEG): container finished" podID="cf2eed7b-85fa-4762-a7a5-3024f050d52e" containerID="a0c0f32d0ea50830c23d62120d428dbd72a5da3103790dc6452ad7fce0eb24da" exitCode=0 Oct 31 01:10:03.165611 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:03.165211 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" event={"ID":"cf2eed7b-85fa-4762-a7a5-3024f050d52e","Type":"ContainerDied","Data":"a0c0f32d0ea50830c23d62120d428dbd72a5da3103790dc6452ad7fce0eb24da"} Oct 31 01:10:03.165611 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:03.165272 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" event={"ID":"cf2eed7b-85fa-4762-a7a5-3024f050d52e","Type":"ContainerStarted","Data":"1b686d0ff9047be212178fc6baa32fe84b2d8a85fcf2b5330f8176539e0c0fc3"} Oct 31 01:10:03.166881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:03.166856 5943 generic.go:356] "Generic (PLEG): container finished" podID="82bb5cda-962d-4b64-855e-29bc388a375e" containerID="0d708b6846d3a97ecedff9a25a5b45355259e34ea599eb686651f54468288628" exitCode=0 Oct 31 01:10:03.166997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:03.166925 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" event={"ID":"82bb5cda-962d-4b64-855e-29bc388a375e","Type":"ContainerDied","Data":"0d708b6846d3a97ecedff9a25a5b45355259e34ea599eb686651f54468288628"} Oct 31 01:10:04.326350 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.326319 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:04.340135 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.340103 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks\" (UniqueName: \"kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks\") pod \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " Oct 31 01:10:04.340378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.340190 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca\") pod \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " Oct 31 01:10:04.340378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.340330 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca\") pod \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\" (UID: \"cf2eed7b-85fa-4762-a7a5-3024f050d52e\") " Oct 31 01:10:04.340598 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.340550 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca" pod "cf2eed7b-85fa-4762-a7a5-3024f050d52e" (UID: "cf2eed7b-85fa-4762-a7a5-3024f050d52e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:04.342674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.342644 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks" pod "cf2eed7b-85fa-4762-a7a5-3024f050d52e" (UID: "cf2eed7b-85fa-4762-a7a5-3024f050d52e"). InnerVolumeSpecName "kube-api-access-wplks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:04.342842 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.342806 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca" pod "cf2eed7b-85fa-4762-a7a5-3024f050d52e" (UID: "cf2eed7b-85fa-4762-a7a5-3024f050d52e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:04.372433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.372378 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:04.441082 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.440982 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca\") pod \"82bb5cda-962d-4b64-855e-29bc388a375e\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " Oct 31 01:10:04.441082 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.441043 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb\" (UniqueName: \"kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb\") pod \"82bb5cda-962d-4b64-855e-29bc388a375e\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " Oct 31 01:10:04.441347 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.441125 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca\") pod \"82bb5cda-962d-4b64-855e-29bc388a375e\" (UID: \"82bb5cda-962d-4b64-855e-29bc388a375e\") " Oct 31 01:10:04.441751 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.441574 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:04.441751 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.441690 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplks\" (UniqueName: \"kubernetes.io/projected/cf2eed7b-85fa-4762-a7a5-3024f050d52e-kube-api-access-wplks\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:04.441751 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.441709 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2eed7b-85fa-4762-a7a5-3024f050d52e-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:04.441987 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.441849 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca" pod "82bb5cda-962d-4b64-855e-29bc388a375e" (UID: "82bb5cda-962d-4b64-855e-29bc388a375e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:04.444338 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.444313 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca" pod "82bb5cda-962d-4b64-855e-29bc388a375e" (UID: "82bb5cda-962d-4b64-855e-29bc388a375e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:04.444743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.444714 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb" pod "82bb5cda-962d-4b64-855e-29bc388a375e" (UID: "82bb5cda-962d-4b64-855e-29bc388a375e"). InnerVolumeSpecName "kube-api-access-spxbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:04.542571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.542528 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:04.542571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.542565 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spxbb\" (UniqueName: \"kubernetes.io/projected/82bb5cda-962d-4b64-855e-29bc388a375e-kube-api-access-spxbb\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:04.542571 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:04.542575 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bb5cda-962d-4b64-855e-29bc388a375e-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:05.181518 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:05.181485 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" Oct 31 01:10:05.181518 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:05.181508 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r" event={"ID":"cf2eed7b-85fa-4762-a7a5-3024f050d52e","Type":"ContainerDied","Data":"1b686d0ff9047be212178fc6baa32fe84b2d8a85fcf2b5330f8176539e0c0fc3"} Oct 31 01:10:05.181761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:05.181567 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b686d0ff9047be212178fc6baa32fe84b2d8a85fcf2b5330f8176539e0c0fc3" Oct 31 01:10:05.182977 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:05.182960 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" Oct 31 01:10:05.182977 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:05.182962 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh" event={"ID":"82bb5cda-962d-4b64-855e-29bc388a375e","Type":"ContainerDied","Data":"20584d3812880f49c19cf15e81f9d7d483df6c56a9fb73d6a93d89e9c964c980"} Oct 31 01:10:05.183124 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:05.183011 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20584d3812880f49c19cf15e81f9d7d483df6c56a9fb73d6a93d89e9c964c980" Oct 31 01:10:06.138960 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:06.138919 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:07.685227 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:07.685162 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:07.964385 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:07.964300 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:08.114425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:08.114391 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:08.178763 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:08.178722 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws"] Oct 31 01:10:08.228196 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:08.228081 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7abb95-hg2ws"] Oct 31 01:10:08.646363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:08.646316 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:10.040742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:10.040707 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61531f8-a297-4f69-bacf-124265d1dba0" path="/var/lib/kubelet/pods/f61531f8-a297-4f69-bacf-124265d1dba0/volumes" Oct 31 01:10:12.548623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:12.548584 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:12.696323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:12.696275 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:13.152541 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:13.152501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:14.773865 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:14.773824 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:14.926910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:14.926864 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:15.154320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:15.154236 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:17.319744 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:17.319702 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:18.817832 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:18.817788 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:19.051997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:19.051963 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:19.199582 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:19.199487 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:19.320460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:19.320422 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl"] Oct 31 01:10:19.368602 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:19.368567 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e1ca71-h4vhl"] Oct 31 01:10:19.777692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:19.777656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:20.040221 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:20.040127 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6fb808-e46e-43ad-86a9-33798ed2f5cf" path="/var/lib/kubelet/pods/3f6fb808-e46e-43ad-86a9-33798ed2f5cf/volumes" Oct 31 01:10:23.543397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:23.543363 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.176669 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:25.176627 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.450989 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:25.450904 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.595867 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:25.595832 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.725236 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:25.725130 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k"] Oct 31 01:10:25.774460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:25.774423 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfc055-4gm7k"] Oct 31 01:10:26.040229 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.040164 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0af2a45-8b92-4b39-8da1-a1c946ba9efa" path="/var/lib/kubelet/pods/c0af2a45-8b92-4b39-8da1-a1c946ba9efa/volumes" Oct 31 01:10:26.109390 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.109347 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:26.256366 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.256327 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:26.412629 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.412543 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:26.422728 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.422697 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb"] Oct 31 01:10:26.422932 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.422910 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" containerID="cri-o://01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7" gracePeriod=30 Oct 31 01:10:26.595345 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:26.595289 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:10:27.478562 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:10:27.478523 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:10:28.043474 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:28.043438 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:28.194490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:28.194448 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:30.547582 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:30.547542 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:31.690730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.690703 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 01:10:31.758133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758105 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758159 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758242 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758288 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758340 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758385 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758423 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758618 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758471 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:31.758618 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758538 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758628 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq\") pod \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\" (UID: \"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3\") " Oct 31 01:10:31.758791 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758739 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:31.758791 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758749 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:31.758791 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758760 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:31.759004 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.758976 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:31.759108 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.759083 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.759108 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.759112 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.759503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.759125 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.759503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.759135 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.759503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.759147 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.760879 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.760852 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "kube-api-access-hwsrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:31.760983 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.760892 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:31.761216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.761197 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:31.761263 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.761197 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo" pod "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" (UID: "fcc253cb-40fc-4786-9c1b-c1dd555ef4b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:10:31.861102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.861065 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.861102 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.861098 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.861377 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.861115 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:31.861377 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:31.861125 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsrq\" (UniqueName: \"kubernetes.io/projected/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3-kube-api-access-hwsrq\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:32.262095 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.262054 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:32.370544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.370451 5943 generic.go:356] "Generic (PLEG): container finished" podID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerID="01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7" exitCode=0 Oct 31 01:10:32.370544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.370508 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" Oct 31 01:10:32.370544 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.370534 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" event={"ID":"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3","Type":"ContainerDied","Data":"01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7"} Oct 31 01:10:32.370826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.370597 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb" event={"ID":"fcc253cb-40fc-4786-9c1b-c1dd555ef4b3","Type":"ContainerDied","Data":"c84bc0d270c676515d42ac6944803fb248ef0402f6405bba3ea920a87eb8887c"} Oct 31 01:10:32.370826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.370623 5943 scope.go:122] "RemoveContainer" containerID="01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7" Oct 31 01:10:32.380966 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.380947 5943 scope.go:122] "RemoveContainer" containerID="01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7" Oct 31 01:10:32.381255 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:10:32.381233 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7\": container with ID starting with 01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7 not found: ID does not exist" containerID="01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7" Oct 31 01:10:32.381337 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.381262 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7"} err="failed to get container status \"01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7\": rpc error: code = NotFound desc = could not find container \"01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7\": container with ID starting with 01981ce26592e7a631892ec862801e940bb63b6f98d5d6c091fdd8bd22b151c7 not found: ID does not exist" Oct 31 01:10:32.453943 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.453904 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:32.589595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.589558 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb"] Oct 31 01:10:32.597859 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.597828 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-5bklb"] Oct 31 01:10:32.622235 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:32.622130 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:33.524702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.524666 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb"] Oct 31 01:10:33.525397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525377 5943 cpu_manager.go:414] "containerMap: removing container" podUID="82bb5cda-962d-4b64-855e-29bc388a375e" containerName="revoke-keys" Oct 31 01:10:33.525507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525398 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82bb5cda-962d-4b64-855e-29bc388a375e" containerName="revoke-keys" Oct 31 01:10:33.525507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525460 5943 cpu_manager.go:414] "containerMap: removing container" podUID="cf2eed7b-85fa-4762-a7a5-3024f050d52e" containerName="revoke-keys" Oct 31 01:10:33.525507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525469 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2eed7b-85fa-4762-a7a5-3024f050d52e" containerName="revoke-keys" Oct 31 01:10:33.525507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525483 5943 cpu_manager.go:414] "containerMap: removing container" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" Oct 31 01:10:33.525507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525491 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" Oct 31 01:10:33.525774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525620 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="82bb5cda-962d-4b64-855e-29bc388a375e" containerName="revoke-keys" Oct 31 01:10:33.525774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525641 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2eed7b-85fa-4762-a7a5-3024f050d52e" containerName="revoke-keys" Oct 31 01:10:33.525774 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.525652 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" containerName="istio-proxy" Oct 31 01:10:33.530533 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.530510 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.578563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.578524 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb4m2\" (UniqueName: \"kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.578787 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.578587 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.578868 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.578802 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.587058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.587015 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb"] Oct 31 01:10:33.633694 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.633664 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:10:33.633694 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.633685 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:10:33.633694 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.633680 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:10:33.633954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.633761 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:10:33.679575 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.679546 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.679758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.679594 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb4m2\" (UniqueName: \"kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.679758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.679649 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.680393 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.680367 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.682193 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.682157 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.740710 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.740666 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb4m2\" (UniqueName: \"kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:33.844394 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:33.844299 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:34.043682 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:34.043644 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc253cb-40fc-4786-9c1b-c1dd555ef4b3" path="/var/lib/kubelet/pods/fcc253cb-40fc-4786-9c1b-c1dd555ef4b3/volumes" Oct 31 01:10:34.222567 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:34.222537 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb"] Oct 31 01:10:34.224456 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:10:34.224426 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aad135_4509_4413_afdf_a80838a31016.slice/crio-8ad43b60f0fa6ef382fe5f7719cad94ccb00414ce6a02bb635e1e27356b8e299 WatchSource:0}: Error finding container 8ad43b60f0fa6ef382fe5f7719cad94ccb00414ce6a02bb635e1e27356b8e299: Status 404 returned error can't find the container with id 8ad43b60f0fa6ef382fe5f7719cad94ccb00414ce6a02bb635e1e27356b8e299 Oct 31 01:10:34.385679 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:34.385635 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" event={"ID":"c4aad135-4509-4413-afdf-a80838a31016","Type":"ContainerStarted","Data":"8ad43b60f0fa6ef382fe5f7719cad94ccb00414ce6a02bb635e1e27356b8e299"} Oct 31 01:10:35.393854 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:35.393749 5943 generic.go:356] "Generic (PLEG): container finished" podID="c4aad135-4509-4413-afdf-a80838a31016" containerID="0ffa06349455123b32a040f1986135bcc1bd727a6073915c7c6418c7df766bbd" exitCode=0 Oct 31 01:10:35.393854 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:35.393828 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" event={"ID":"c4aad135-4509-4413-afdf-a80838a31016","Type":"ContainerDied","Data":"0ffa06349455123b32a040f1986135bcc1bd727a6073915c7c6418c7df766bbd"} Oct 31 01:10:35.498810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:35.498763 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:36.559269 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.559239 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:36.611727 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.611685 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca\") pod \"c4aad135-4509-4413-afdf-a80838a31016\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " Oct 31 01:10:36.611942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.611825 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca\") pod \"c4aad135-4509-4413-afdf-a80838a31016\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " Oct 31 01:10:36.611942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.611922 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2\" (UniqueName: \"kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2\") pod \"c4aad135-4509-4413-afdf-a80838a31016\" (UID: \"c4aad135-4509-4413-afdf-a80838a31016\") " Oct 31 01:10:36.612283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.612255 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca" pod "c4aad135-4509-4413-afdf-a80838a31016" (UID: "c4aad135-4509-4413-afdf-a80838a31016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:36.613989 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.613965 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca" pod "c4aad135-4509-4413-afdf-a80838a31016" (UID: "c4aad135-4509-4413-afdf-a80838a31016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:36.614139 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.614115 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2" pod "c4aad135-4509-4413-afdf-a80838a31016" (UID: "c4aad135-4509-4413-afdf-a80838a31016"). InnerVolumeSpecName "kube-api-access-tb4m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:36.713238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.713122 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb4m2\" (UniqueName: \"kubernetes.io/projected/c4aad135-4509-4413-afdf-a80838a31016-kube-api-access-tb4m2\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:36.713238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.713156 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4aad135-4509-4413-afdf-a80838a31016-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:36.713238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:36.713168 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4aad135-4509-4413-afdf-a80838a31016-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.409369 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:37.409326 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" event={"ID":"c4aad135-4509-4413-afdf-a80838a31016","Type":"ContainerDied","Data":"8ad43b60f0fa6ef382fe5f7719cad94ccb00414ce6a02bb635e1e27356b8e299"} Oct 31 01:10:37.409369 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:37.409360 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb" Oct 31 01:10:37.409601 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:37.409388 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad43b60f0fa6ef382fe5f7719cad94ccb00414ce6a02bb635e1e27356b8e299" Oct 31 01:10:40.600693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:40.600646 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:42.449587 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:42.449507 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:10:42.973837 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:42.973780 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:44.199919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:44.199880 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:44.358869 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:44.358814 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:44.527002 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:44.526961 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:44.611532 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:44.611493 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn"] Oct 31 01:10:44.842626 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:44.842543 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.091400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:45.091363 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.238816 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:45.238726 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.868959 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:45.868918 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:46.015716 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:46.015677 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:48.441091 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:48.441053 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:49.612002 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:49.611952 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerName="payload-processing" containerID="cri-o://b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278" gracePeriod=25 Oct 31 01:10:50.038126 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:50.038082 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.537802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:51.537767 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.748233 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:51.748197 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.900732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:51.900647 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:52.041754 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:52.041727 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx"] Oct 31 01:10:52.084748 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:52.084691 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-869cca-fc8wx"] Oct 31 01:10:52.495043 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:52.495004 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:54.043040 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:54.043003 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51643a7c-c4bc-4851-9dad-4880d094c2dc" path="/var/lib/kubelet/pods/51643a7c-c4bc-4851-9dad-4880d094c2dc/volumes" Oct 31 01:10:56.843541 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:56.843490 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:57.475767 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:10:57.475731 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:10:58.486521 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:58.486480 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:10:58.863250 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:58.863205 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:59.061609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:59.061567 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:59.232908 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:10:59.232818 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:00.336409 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:00.336373 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:00.479989 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:00.479946 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:01.483042 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:01.483007 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:01.649317 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:01.649286 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:06.128362 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:06.128320 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:11:06.441251 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:06.441136 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:08.486133 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:08.486087 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:11:09.874076 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:09.874009 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 01:11:09.964888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:09.964852 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume\") pod \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " Oct 31 01:11:09.965067 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:09.964898 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q\" (UniqueName: \"kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q\") pod \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\" (UID: \"5c1e350c-f3a8-4cee-91ad-7864f9ad7648\") " Oct 31 01:11:09.965283 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:09.965261 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume" pod "5c1e350c-f3a8-4cee-91ad-7864f9ad7648" (UID: "5c1e350c-f3a8-4cee-91ad-7864f9ad7648"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:09.967138 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:09.967116 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q" pod "5c1e350c-f3a8-4cee-91ad-7864f9ad7648" (UID: "5c1e350c-f3a8-4cee-91ad-7864f9ad7648"). InnerVolumeSpecName "kube-api-access-7rn7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:10.066692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.066657 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:10.066692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.066692 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rn7q\" (UniqueName: \"kubernetes.io/projected/5c1e350c-f3a8-4cee-91ad-7864f9ad7648-kube-api-access-7rn7q\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:10.655016 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.654976 5943 generic.go:356] "Generic (PLEG): container finished" podID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerID="b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278" exitCode=0 Oct 31 01:11:10.655257 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.655018 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" event={"ID":"5c1e350c-f3a8-4cee-91ad-7864f9ad7648","Type":"ContainerDied","Data":"b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278"} Oct 31 01:11:10.655257 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.655035 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" Oct 31 01:11:10.655257 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.655074 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn" event={"ID":"5c1e350c-f3a8-4cee-91ad-7864f9ad7648","Type":"ContainerDied","Data":"17a9b1288e646d79e0a2249be6d632b6535f27e55b982be59a76c8ffdb068a1e"} Oct 31 01:11:10.655257 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.655096 5943 scope.go:122] "RemoveContainer" containerID="b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278" Oct 31 01:11:10.665111 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.665095 5943 scope.go:122] "RemoveContainer" containerID="b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278" Oct 31 01:11:10.665406 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:10.665388 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278\": container with ID starting with b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278 not found: ID does not exist" containerID="b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278" Oct 31 01:11:10.665460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.665417 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278"} err="failed to get container status \"b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278\": rpc error: code = NotFound desc = could not find container \"b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278\": container with ID starting with b9c575487992b84cc7acdaf8cecbbc94246c05a28c48b8f2dc408c8065f57278 not found: ID does not exist" Oct 31 01:11:10.818874 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.818839 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn"] Oct 31 01:11:10.868106 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:10.868071 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c6b3b9-68f9dbfb57-fnsnn"] Oct 31 01:11:11.250743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:11.250709 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:12.040843 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:12.040809 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" path="/var/lib/kubelet/pods/5c1e350c-f3a8-4cee-91ad-7864f9ad7648/volumes" Oct 31 01:11:16.052602 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:16.052570 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:20.199924 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:20.199835 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:21.717333 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:21.717295 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:21.957065 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:21.957027 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:22.103966 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:22.103920 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:22.209903 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:22.209868 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb"] Oct 31 01:11:22.259412 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:22.259381 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c6b3b9-txdpb"] Oct 31 01:11:22.650154 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:22.650115 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:24.040988 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:24.040946 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4aad135-4509-4413-afdf-a80838a31016" path="/var/lib/kubelet/pods/c4aad135-4509-4413-afdf-a80838a31016/volumes" Oct 31 01:11:27.205998 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:27.205966 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:27.476900 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:27.476808 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:11:28.013841 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:28.013790 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:11:29.301842 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:29.301801 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:29.498808 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:29.498761 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:29.733313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:29.733220 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449"] Oct 31 01:11:29.733474 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:29.733447 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" containerID="cri-o://aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6" gracePeriod=30 Oct 31 01:11:29.789376 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:29.789341 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:30.210704 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.210656 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:30.691042 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.690999 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw"] Oct 31 01:11:30.691517 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.691503 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerName="payload-processing" Oct 31 01:11:30.691563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.691518 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerName="payload-processing" Oct 31 01:11:30.691563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.691550 5943 cpu_manager.go:414] "containerMap: removing container" podUID="c4aad135-4509-4413-afdf-a80838a31016" containerName="revoke-keys" Oct 31 01:11:30.691563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.691555 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4aad135-4509-4413-afdf-a80838a31016" containerName="revoke-keys" Oct 31 01:11:30.691659 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.691623 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1e350c-f3a8-4cee-91ad-7864f9ad7648" containerName="payload-processing" Oct 31 01:11:30.691659 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.691638 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4aad135-4509-4413-afdf-a80838a31016" containerName="revoke-keys" Oct 31 01:11:30.695019 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.695001 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.747125 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.747091 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:11:30.792457 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.792417 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.792684 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.792578 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr8d\" (UniqueName: \"kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.792684 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.792619 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.796761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.796725 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:11:30.796761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.796752 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:11:30.796949 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.796796 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:11:30.799207 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.799171 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw"] Oct 31 01:11:30.893418 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.893378 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.893607 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.893458 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr8d\" (UniqueName: \"kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.893607 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.893483 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.894068 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.894035 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.895937 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.895908 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:30.954558 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:30.954485 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr8d\" (UniqueName: \"kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d\") pod \"maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:31.008551 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:31.008519 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:31.232726 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:31.232639 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:31.377977 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:31.377932 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:31.387295 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:31.387255 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw"] Oct 31 01:11:31.391115 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:11:31.391089 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa1df21_ad2e_48e7_88ff_91cd0f9c9a55.slice/crio-5af1df25ec188c47b432ca65cd52edc9e658d4fb9ebffc54463087336665ddc5 WatchSource:0}: Error finding container 5af1df25ec188c47b432ca65cd52edc9e658d4fb9ebffc54463087336665ddc5: Status 404 returned error can't find the container with id 5af1df25ec188c47b432ca65cd52edc9e658d4fb9ebffc54463087336665ddc5 Oct 31 01:11:31.793508 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:31.793474 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" event={"ID":"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55","Type":"ContainerStarted","Data":"5af1df25ec188c47b432ca65cd52edc9e658d4fb9ebffc54463087336665ddc5"} Oct 31 01:11:32.195503 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:32.195468 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:32.801475 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:32.801389 5943 generic.go:356] "Generic (PLEG): container finished" podID="cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" containerID="c7ce183fc2d204cb3b40ff3309017684d725caff4d6c01c462e6f3ea7511de60" exitCode=0 Oct 31 01:11:32.801857 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:32.801474 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" event={"ID":"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55","Type":"ContainerDied","Data":"c7ce183fc2d204cb3b40ff3309017684d725caff4d6c01c462e6f3ea7511de60"} Oct 31 01:11:33.953506 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:33.953480 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:34.132693 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.132603 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca\") pod \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " Oct 31 01:11:34.132857 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.132708 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d\" (UniqueName: \"kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d\") pod \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " Oct 31 01:11:34.132857 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.132799 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca\") pod \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\" (UID: \"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55\") " Oct 31 01:11:34.133270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.133246 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca" pod "cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" (UID: "cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:34.135651 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.135625 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d" pod "cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" (UID: "cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55"). InnerVolumeSpecName "kube-api-access-tgr8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:34.135748 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.135625 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca" pod "cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" (UID: "cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:34.234553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.234516 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:34.234553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.234550 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgr8d\" (UniqueName: \"kubernetes.io/projected/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-kube-api-access-tgr8d\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:34.234553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.234561 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:34.816103 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.816061 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" event={"ID":"cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55","Type":"ContainerDied","Data":"5af1df25ec188c47b432ca65cd52edc9e658d4fb9ebffc54463087336665ddc5"} Oct 31 01:11:34.816103 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.816104 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw" Oct 31 01:11:34.816356 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.816162 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af1df25ec188c47b432ca65cd52edc9e658d4fb9ebffc54463087336665ddc5" Oct 31 01:11:34.989608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:34.989583 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 01:11:35.142971 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.142940 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143018 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143054 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143222 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143151 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143252 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143282 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143344 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143411 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143439 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:35.143658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143472 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket\") pod \"0a4584d7-e974-443c-955c-14e36f5a14fb\" (UID: \"0a4584d7-e974-443c-955c-14e36f5a14fb\") " Oct 31 01:11:35.143954 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.143933 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4584d7-e974-443c-955c-14e36f5a14fb-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.144150 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.144011 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:35.144272 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.144231 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:35.144359 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.144339 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:35.144610 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.144591 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:35.146524 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.146502 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "kube-api-access-f6q2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:35.146631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.146549 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:35.146631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.146586 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:11:35.148025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.148007 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token" pod "0a4584d7-e974-443c-955c-14e36f5a14fb" (UID: "0a4584d7-e974-443c-955c-14e36f5a14fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:35.174036 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.174000 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl"] Oct 31 01:11:35.185450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.185421 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7"] Oct 31 01:11:35.244848 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244810 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.244848 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244839 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.244848 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244850 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.245100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244859 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.245100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244867 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.245100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244876 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.245100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244884 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4584d7-e974-443c-955c-14e36f5a14fb-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.245100 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.244893 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6q2m\" (UniqueName: \"kubernetes.io/projected/0a4584d7-e974-443c-955c-14e36f5a14fb-kube-api-access-f6q2m\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:35.367756 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.367716 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:35.824552 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.824519 5943 generic.go:356] "Generic (PLEG): container finished" podID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerID="aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6" exitCode=0 Oct 31 01:11:35.824790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.824591 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" Oct 31 01:11:35.824790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.824604 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" event={"ID":"0a4584d7-e974-443c-955c-14e36f5a14fb","Type":"ContainerDied","Data":"aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6"} Oct 31 01:11:35.824790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.824678 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449" event={"ID":"0a4584d7-e974-443c-955c-14e36f5a14fb","Type":"ContainerDied","Data":"5f00ddff1ee13481bb88b71d36788113549d69077c5d970796f9b1de9b947f87"} Oct 31 01:11:35.824790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.824710 5943 scope.go:122] "RemoveContainer" containerID="aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6" Oct 31 01:11:35.836235 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.836213 5943 scope.go:122] "RemoveContainer" containerID="9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71" Oct 31 01:11:35.845243 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.845226 5943 scope.go:122] "RemoveContainer" containerID="aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6" Oct 31 01:11:35.845520 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:35.845501 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6\": container with ID starting with aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6 not found: ID does not exist" containerID="aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6" Oct 31 01:11:35.845595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.845529 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6"} err="failed to get container status \"aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6\": rpc error: code = NotFound desc = could not find container \"aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6\": container with ID starting with aa77aaa2a09b268acba5ea926350feadb2e3dcaffc2da2162a383daa2abfb9f6 not found: ID does not exist" Oct 31 01:11:35.845595 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.845547 5943 scope.go:122] "RemoveContainer" containerID="9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71" Oct 31 01:11:35.845781 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:35.845766 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71\": container with ID starting with 9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71 not found: ID does not exist" containerID="9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71" Oct 31 01:11:35.845818 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.845785 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71"} err="failed to get container status \"9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71\": rpc error: code = NotFound desc = could not find container \"9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71\": container with ID starting with 9807249864c804cf2996609d47dd88cd5a9cfbac1c38e4d6295c4c88413d8c71 not found: ID does not exist" Oct 31 01:11:35.991664 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:35.991631 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449"] Oct 31 01:11:36.041028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:36.040996 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c6b3b9-openshift-default-bd978587-sj449"] Oct 31 01:11:37.184815 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:37.184773 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:37.884579 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:37.884548 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:38.050215 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:38.050146 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" path="/var/lib/kubelet/pods/0a4584d7-e974-443c-955c-14e36f5a14fb/volumes" Oct 31 01:11:42.041407 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:42.041380 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:42.076470 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:42.076440 5943 scope.go:122] "RemoveContainer" containerID="93b61cecc1a07abeea4c6437a544e110ee5836e0f05e786cbc51bf8d77d09a9e" Oct 31 01:11:42.189326 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:42.189291 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:44.337562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:44.337517 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:46.785858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:46.785773 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:50.195986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.195917 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" containerID="cri-o://ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843" gracePeriod=45 Oct 31 01:11:50.205566 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.205514 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" containerID="cri-o://7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519" gracePeriod=45 Oct 31 01:11:50.434097 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.434061 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:11:50.483660 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.483637 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 01:11:50.486805 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.486780 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 01:11:50.612091 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612056 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs\") pod \"1ff896c2-e62d-4484-826d-526bcd6adc17\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " Oct 31 01:11:50.612304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612119 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir\") pod \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " Oct 31 01:11:50.612304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612154 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir\") pod \"1ff896c2-e62d-4484-826d-526bcd6adc17\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " Oct 31 01:11:50.612304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612208 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache\") pod \"1ff896c2-e62d-4484-826d-526bcd6adc17\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " Oct 31 01:11:50.612304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612278 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home\") pod \"1ff896c2-e62d-4484-826d-526bcd6adc17\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612310 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache\") pod \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612360 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm\") pod \"1ff896c2-e62d-4484-826d-526bcd6adc17\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612434 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm\") pod \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612462 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6\" (UniqueName: \"kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6\") pod \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612469 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir" pod "1ff896c2-e62d-4484-826d-526bcd6adc17" (UID: "1ff896c2-e62d-4484-826d-526bcd6adc17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612494 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home\") pod \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " Oct 31 01:11:50.612543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612493 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache" pod "1ff896c2-e62d-4484-826d-526bcd6adc17" (UID: "1ff896c2-e62d-4484-826d-526bcd6adc17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.613077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612557 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs\") pod \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\" (UID: \"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5\") " Oct 31 01:11:50.613077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612592 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v\" (UniqueName: \"kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v\") pod \"1ff896c2-e62d-4484-826d-526bcd6adc17\" (UID: \"1ff896c2-e62d-4484-826d-526bcd6adc17\") " Oct 31 01:11:50.613077 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.612981 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache" pod "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" (UID: "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.613281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.613107 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.613281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.613127 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.613281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.613141 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.613449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.613337 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home" pod "1ff896c2-e62d-4484-826d-526bcd6adc17" (UID: "1ff896c2-e62d-4484-826d-526bcd6adc17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.613647 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.613620 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home" pod "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" (UID: "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.613961 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.613938 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir" pod "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" (UID: "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.615243 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.615213 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs" pod "1ff896c2-e62d-4484-826d-526bcd6adc17" (UID: "1ff896c2-e62d-4484-826d-526bcd6adc17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:50.615349 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.615253 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm" pod "1ff896c2-e62d-4484-826d-526bcd6adc17" (UID: "1ff896c2-e62d-4484-826d-526bcd6adc17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.615424 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.615354 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm" pod "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" (UID: "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.615646 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.615627 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs" pod "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" (UID: "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:50.615811 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.615786 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v" pod "1ff896c2-e62d-4484-826d-526bcd6adc17" (UID: "1ff896c2-e62d-4484-826d-526bcd6adc17"). InnerVolumeSpecName "kube-api-access-p7x8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:50.615988 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.615966 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6" pod "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" (UID: "5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5"). InnerVolumeSpecName "kube-api-access-hfpb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714100 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714132 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7x8v\" (UniqueName: \"kubernetes.io/projected/1ff896c2-e62d-4484-826d-526bcd6adc17-kube-api-access-p7x8v\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714143 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff896c2-e62d-4484-826d-526bcd6adc17-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714151 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714161 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714168 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff896c2-e62d-4484-826d-526bcd6adc17-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714191 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714199 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfpb6\" (UniqueName: \"kubernetes.io/projected/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-kube-api-access-hfpb6\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.714197 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.714206 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.924467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.924426 5943 generic.go:356] "Generic (PLEG): container finished" podID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerID="ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843" exitCode=0 Oct 31 01:11:50.924674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.924476 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" Oct 31 01:11:50.924674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.924513 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" event={"ID":"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5","Type":"ContainerDied","Data":"ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843"} Oct 31 01:11:50.924674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.924560 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl" event={"ID":"5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5","Type":"ContainerDied","Data":"4de89e99e7a34655eba41a0e96657733197958eb01581ab08068d67041480efe"} Oct 31 01:11:50.924674 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.924578 5943 scope.go:122] "RemoveContainer" containerID="ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843" Oct 31 01:11:50.926439 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.926374 5943 generic.go:356] "Generic (PLEG): container finished" podID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerID="7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519" exitCode=0 Oct 31 01:11:50.926439 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.926421 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" Oct 31 01:11:50.926631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.926474 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" event={"ID":"1ff896c2-e62d-4484-826d-526bcd6adc17","Type":"ContainerDied","Data":"7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519"} Oct 31 01:11:50.926631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.926524 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7" event={"ID":"1ff896c2-e62d-4484-826d-526bcd6adc17","Type":"ContainerDied","Data":"85b311fd87be05b8f51dab024e056c7ff6043fe19c1b72d1ed0734fb8fb4fec8"} Oct 31 01:11:50.935749 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.935655 5943 scope.go:122] "RemoveContainer" containerID="ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843" Oct 31 01:11:50.936017 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:50.935992 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843\": container with ID starting with ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843 not found: ID does not exist" containerID="ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843" Oct 31 01:11:50.936109 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.936028 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843"} err="failed to get container status \"ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843\": rpc error: code = NotFound desc = could not find container \"ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843\": container with ID starting with ad279bc2e41173f3de393c4ee7b4a0c7c935ea2f9513562fbd2f1d8b99d21843 not found: ID does not exist" Oct 31 01:11:50.936109 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.936051 5943 scope.go:122] "RemoveContainer" containerID="7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519" Oct 31 01:11:50.945128 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.945108 5943 scope.go:122] "RemoveContainer" containerID="7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519" Oct 31 01:11:50.945418 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:50.945398 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519\": container with ID starting with 7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519 not found: ID does not exist" containerID="7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519" Oct 31 01:11:50.945485 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:50.945424 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519"} err="failed to get container status \"7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519\": rpc error: code = NotFound desc = could not find container \"7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519\": container with ID starting with 7c56726d8e1a86c937eb3e05c129a0e247dc01d421f6b64b2efbbb9c6a340519 not found: ID does not exist" Oct 31 01:11:51.088020 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:51.087984 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7"] Oct 31 01:11:51.091441 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:51.091416 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/premium-w0-10a7aa-kserve-cd4d5979-tpcx7"] Oct 31 01:11:51.199711 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:51.199675 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl"] Oct 31 01:11:51.204956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:51.204922 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-w0-10a7aa-kserve-d879cd794-sswcl"] Oct 31 01:11:51.536903 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:51.536865 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:52.039883 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:52.039855 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" path="/var/lib/kubelet/pods/1ff896c2-e62d-4484-826d-526bcd6adc17/volumes" Oct 31 01:11:52.040294 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:52.040280 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" path="/var/lib/kubelet/pods/5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5/volumes" Oct 31 01:11:56.296900 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:11:56.296861 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:57.475152 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:11:57.475115 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:12:01.050082 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:01.050036 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:01.837539 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:01.837499 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:05.204020 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.203982 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl"] Oct 31 01:12:05.204668 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204646 5943 cpu_manager.go:414] "containerMap: removing container" podUID="cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" containerName="revoke-keys" Oct 31 01:12:05.204760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204670 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" containerName="revoke-keys" Oct 31 01:12:05.204760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204694 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" Oct 31 01:12:05.204760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204703 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" Oct 31 01:12:05.204760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204739 5943 cpu_manager.go:414] "containerMap: removing container" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" Oct 31 01:12:05.204760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204748 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204772 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204780 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204792 5943 cpu_manager.go:414] "containerMap: removing container" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204800 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204940 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" containerName="revoke-keys" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204953 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204969 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f683ac1-9ac3-4174-b3c9-9d8a8a1396e5" containerName="main" Oct 31 01:12:05.205025 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.204980 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff896c2-e62d-4484-826d-526bcd6adc17" containerName="main" Oct 31 01:12:05.210167 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.210146 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.256702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.256673 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp"] Oct 31 01:12:05.257228 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.257214 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4584d7-e974-443c-955c-14e36f5a14fb" containerName="istio-proxy" Oct 31 01:12:05.260504 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.260488 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.304299 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.304266 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:12:05.313062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.313021 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:12:05.313062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.313038 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:12:05.313345 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.313077 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:12:05.315285 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.315266 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl"] Oct 31 01:12:05.319578 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.319540 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp"] Oct 31 01:12:05.368745 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.368686 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flrd\" (UniqueName: \"kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.368948 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.368892 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8khx\" (UniqueName: \"kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.369023 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.368944 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.369084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.369024 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.369084 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.369077 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.369218 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.369107 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.470507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.470415 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.470507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.470467 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flrd\" (UniqueName: \"kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.470737 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.470539 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8khx\" (UniqueName: \"kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.470737 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.470570 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.470737 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.470598 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.470737 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.470651 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.471332 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.471306 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.471427 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.471310 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.473099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.473079 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.473274 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.473258 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.533519 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.533488 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8khx\" (UniqueName: \"kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx\") pod \"maas-api-revoke-sub-e024b754-6w9dp\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.533678 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.533642 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flrd\" (UniqueName: \"kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd\") pod \"maas-api-revoke-sub-06a462b8-6tbrl\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.575581 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.575543 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:05.823131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.823093 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:05.949624 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:05.949550 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp"] Oct 31 01:12:05.951998 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:12:05.951975 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e2439b_3459_49c6_9f15_46e960ca98b1.slice/crio-5b401968c6fa021ae010f50ebe39867f0d18679961aa8ae50993801789fadb38 WatchSource:0}: Error finding container 5b401968c6fa021ae010f50ebe39867f0d18679961aa8ae50993801789fadb38: Status 404 returned error can't find the container with id 5b401968c6fa021ae010f50ebe39867f0d18679961aa8ae50993801789fadb38 Oct 31 01:12:06.030852 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:06.030816 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" event={"ID":"15e2439b-3459-49c6-9f15-46e960ca98b1","Type":"ContainerStarted","Data":"5b401968c6fa021ae010f50ebe39867f0d18679961aa8ae50993801789fadb38"} Oct 31 01:12:06.051428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:06.051393 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:06.200705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:06.200647 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl"] Oct 31 01:12:06.202634 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:12:06.202602 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc2003a_d129_4d24_9085_503fd6974526.slice/crio-31996023761a28d908bef63b635f7638b3454cf8e54e2903847a8444ca809c14 WatchSource:0}: Error finding container 31996023761a28d908bef63b635f7638b3454cf8e54e2903847a8444ca809c14: Status 404 returned error can't find the container with id 31996023761a28d908bef63b635f7638b3454cf8e54e2903847a8444ca809c14 Oct 31 01:12:07.039293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:07.039199 5943 generic.go:356] "Generic (PLEG): container finished" podID="15e2439b-3459-49c6-9f15-46e960ca98b1" containerID="bfa88109b95bd06ae06c2b29e8f79b11e8f427f1269bdef46204f539caa7ca58" exitCode=0 Oct 31 01:12:07.039293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:07.039272 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" event={"ID":"15e2439b-3459-49c6-9f15-46e960ca98b1","Type":"ContainerDied","Data":"bfa88109b95bd06ae06c2b29e8f79b11e8f427f1269bdef46204f539caa7ca58"} Oct 31 01:12:07.041007 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:07.040984 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" event={"ID":"8fc2003a-d129-4d24-9085-503fd6974526","Type":"ContainerStarted","Data":"2ba771fb0f583a10bb85df6748c326b529ed064fce219d53c789d8411c848e9d"} Oct 31 01:12:07.041120 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:07.041023 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" event={"ID":"8fc2003a-d129-4d24-9085-503fd6974526","Type":"ContainerStarted","Data":"31996023761a28d908bef63b635f7638b3454cf8e54e2903847a8444ca809c14"} Oct 31 01:12:07.270802 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:07.270747 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" podStartSLOduration=2.2707266170000002 podStartE2EDuration="2.270726617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:07.269241452 +0000 UTC m=+4107.362531662" watchObservedRunningTime="2026-10-31 01:12:07.270726617 +0000 UTC m=+4107.364016805" Oct 31 01:12:08.049089 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.049057 5943 generic.go:356] "Generic (PLEG): container finished" podID="8fc2003a-d129-4d24-9085-503fd6974526" containerID="2ba771fb0f583a10bb85df6748c326b529ed064fce219d53c789d8411c848e9d" exitCode=0 Oct 31 01:12:08.049542 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.049138 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" event={"ID":"8fc2003a-d129-4d24-9085-503fd6974526","Type":"ContainerDied","Data":"2ba771fb0f583a10bb85df6748c326b529ed064fce219d53c789d8411c848e9d"} Oct 31 01:12:08.083272 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.083236 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:12:08.193011 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.192987 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:08.197326 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.197295 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx\" (UniqueName: \"kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx\") pod \"15e2439b-3459-49c6-9f15-46e960ca98b1\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " Oct 31 01:12:08.197497 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.197355 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca\") pod \"15e2439b-3459-49c6-9f15-46e960ca98b1\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " Oct 31 01:12:08.197497 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.197447 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca\") pod \"15e2439b-3459-49c6-9f15-46e960ca98b1\" (UID: \"15e2439b-3459-49c6-9f15-46e960ca98b1\") " Oct 31 01:12:08.197712 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.197682 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca" pod "15e2439b-3459-49c6-9f15-46e960ca98b1" (UID: "15e2439b-3459-49c6-9f15-46e960ca98b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:08.197856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.197832 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:08.199789 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.199750 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx" pod "15e2439b-3459-49c6-9f15-46e960ca98b1" (UID: "15e2439b-3459-49c6-9f15-46e960ca98b1"). InnerVolumeSpecName "kube-api-access-q8khx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:08.200013 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.199991 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca" pod "15e2439b-3459-49c6-9f15-46e960ca98b1" (UID: "15e2439b-3459-49c6-9f15-46e960ca98b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:08.298649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.298613 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e2439b-3459-49c6-9f15-46e960ca98b1-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:08.298649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:08.298645 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8khx\" (UniqueName: \"kubernetes.io/projected/15e2439b-3459-49c6-9f15-46e960ca98b1-kube-api-access-q8khx\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:09.056661 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.056627 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" Oct 31 01:12:09.056661 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.056648 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp" event={"ID":"15e2439b-3459-49c6-9f15-46e960ca98b1","Type":"ContainerDied","Data":"5b401968c6fa021ae010f50ebe39867f0d18679961aa8ae50993801789fadb38"} Oct 31 01:12:09.057137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.056708 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b401968c6fa021ae010f50ebe39867f0d18679961aa8ae50993801789fadb38" Oct 31 01:12:09.109472 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:12:09.109441 5943 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e2439b_3459_49c6_9f15_46e960ca98b1.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:12:09.200164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.200131 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:09.205375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.205345 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca\") pod \"8fc2003a-d129-4d24-9085-503fd6974526\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " Oct 31 01:12:09.205535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.205424 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca\") pod \"8fc2003a-d129-4d24-9085-503fd6974526\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " Oct 31 01:12:09.205535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.205475 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd\" (UniqueName: \"kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd\") pod \"8fc2003a-d129-4d24-9085-503fd6974526\" (UID: \"8fc2003a-d129-4d24-9085-503fd6974526\") " Oct 31 01:12:09.205821 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.205787 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca" pod "8fc2003a-d129-4d24-9085-503fd6974526" (UID: "8fc2003a-d129-4d24-9085-503fd6974526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:09.207616 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.207593 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca" pod "8fc2003a-d129-4d24-9085-503fd6974526" (UID: "8fc2003a-d129-4d24-9085-503fd6974526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:09.207743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.207724 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd" pod "8fc2003a-d129-4d24-9085-503fd6974526" (UID: "8fc2003a-d129-4d24-9085-503fd6974526"). InnerVolumeSpecName "kube-api-access-5flrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:09.306732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.306653 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:09.306732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.306678 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flrd\" (UniqueName: \"kubernetes.io/projected/8fc2003a-d129-4d24-9085-503fd6974526-kube-api-access-5flrd\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:09.306732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:09.306687 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc2003a-d129-4d24-9085-503fd6974526-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:10.064389 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:10.064357 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" Oct 31 01:12:10.064759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:10.064389 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl" event={"ID":"8fc2003a-d129-4d24-9085-503fd6974526","Type":"ContainerDied","Data":"31996023761a28d908bef63b635f7638b3454cf8e54e2903847a8444ca809c14"} Oct 31 01:12:10.064759 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:10.064439 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31996023761a28d908bef63b635f7638b3454cf8e54e2903847a8444ca809c14" Oct 31 01:12:10.860640 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:10.860605 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:11.661698 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:11.661663 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:11.815551 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:11.815516 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:13.561398 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:13.561365 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:15.626540 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:15.626501 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:19.728589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:19.728551 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:19.924662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:19.924620 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:20.351714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:20.351677 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:25.144773 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:25.144734 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:27.475591 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:12:27.475554 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:12:29.880354 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:29.880314 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:34.458700 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:34.458658 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:12:34.633342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:34.633303 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:37.487143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:37.487105 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:37.639085 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:37.639045 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:38.576944 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:38.576913 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:38.724676 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:38.724616 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:38.875548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:38.875464 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:39.413979 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.413941 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95"] Oct 31 01:12:39.414837 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.414804 5943 cpu_manager.go:414] "containerMap: removing container" podUID="8fc2003a-d129-4d24-9085-503fd6974526" containerName="revoke-keys" Oct 31 01:12:39.414837 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.414832 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc2003a-d129-4d24-9085-503fd6974526" containerName="revoke-keys" Oct 31 01:12:39.415017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.414882 5943 cpu_manager.go:414] "containerMap: removing container" podUID="15e2439b-3459-49c6-9f15-46e960ca98b1" containerName="revoke-keys" Oct 31 01:12:39.415017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.414889 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e2439b-3459-49c6-9f15-46e960ca98b1" containerName="revoke-keys" Oct 31 01:12:39.415017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.414973 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc2003a-d129-4d24-9085-503fd6974526" containerName="revoke-keys" Oct 31 01:12:39.415017 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.414987 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e2439b-3459-49c6-9f15-46e960ca98b1" containerName="revoke-keys" Oct 31 01:12:39.418382 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.418364 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.523325 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.523291 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:12:39.523325 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.523309 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:12:39.523325 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.523309 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:12:39.523594 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.523297 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:12:39.525790 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.525766 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95"] Oct 31 01:12:39.534018 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.533995 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.534135 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.534023 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.534135 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.534067 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7lx\" (UniqueName: \"kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.634742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.634708 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.634742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.634745 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.635265 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.634797 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7lx\" (UniqueName: \"kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.635398 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.635377 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.637281 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.637250 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.696083 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.695998 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7lx\" (UniqueName: \"kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx\") pod \"maas-api-revoke-sub-46d4bcfd-9lq95\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:39.731701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:39.731671 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:40.110494 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.110468 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95"] Oct 31 01:12:40.112353 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:12:40.112312 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aba2032_5797_4934_ab3b_42c7095d6c0a.slice/crio-f5bcfae43368f42e607bff5c3a3ca31dab97d1d4228ef29dbd2abda195813d8c WatchSource:0}: Error finding container f5bcfae43368f42e607bff5c3a3ca31dab97d1d4228ef29dbd2abda195813d8c: Status 404 returned error can't find the container with id f5bcfae43368f42e607bff5c3a3ca31dab97d1d4228ef29dbd2abda195813d8c Oct 31 01:12:40.270696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.270660 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" event={"ID":"1aba2032-5797-4934-ab3b-42c7095d6c0a","Type":"ContainerStarted","Data":"f5bcfae43368f42e607bff5c3a3ca31dab97d1d4228ef29dbd2abda195813d8c"} Oct 31 01:12:40.359145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.359112 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:40.659233 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.659189 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:40.808326 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.808294 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:40.923163 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.923130 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw"] Oct 31 01:12:40.972708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:40.972672 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10a7aa-tdbgw"] Oct 31 01:12:41.279691 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:41.279656 5943 generic.go:356] "Generic (PLEG): container finished" podID="1aba2032-5797-4934-ab3b-42c7095d6c0a" containerID="d38707ac6ae83bcd17dcb5687fc773c91ee8fe2cb6e2b7a38cecaa80daa5769e" exitCode=0 Oct 31 01:12:41.279890 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:41.279747 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" event={"ID":"1aba2032-5797-4934-ab3b-42c7095d6c0a","Type":"ContainerDied","Data":"d38707ac6ae83bcd17dcb5687fc773c91ee8fe2cb6e2b7a38cecaa80daa5769e"} Oct 31 01:12:41.343624 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:41.343584 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:42.040229 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.040193 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55" path="/var/lib/kubelet/pods/cfa1df21-ad2e-48e7-88ff-91cd0f9c9a55/volumes" Oct 31 01:12:42.118848 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.118815 5943 scope.go:122] "RemoveContainer" containerID="c2ab84925101adbb6867cdd363284310b79e664aa3922ff5c9bc356a9374e37a" Oct 31 01:12:42.128492 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.128471 5943 scope.go:122] "RemoveContainer" containerID="1af40e9fb3fdc71f7c45bb9a45db2f950b6cf5aaeb4994c307ef56d136d5b192" Oct 31 01:12:42.137005 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.136987 5943 scope.go:122] "RemoveContainer" containerID="f4a69942693db4ecf5a1dc14477e3591da30b229bc3d3bfdfedea0eaa3b4ea15" Oct 31 01:12:42.145324 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.145304 5943 scope.go:122] "RemoveContainer" containerID="0b2facde85b287e121208b6458ebe1373b2a11ca79ad8b62792fbbd7b3af0cfe" Oct 31 01:12:42.154410 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.154393 5943 scope.go:122] "RemoveContainer" containerID="b3ec7b9734f0fe6e1161325f464849f10255a704d1e144c24c896c9c5064fda4" Oct 31 01:12:42.428074 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.428049 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:42.462934 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.462899 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx\" (UniqueName: \"kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx\") pod \"1aba2032-5797-4934-ab3b-42c7095d6c0a\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " Oct 31 01:12:42.463142 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.462963 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca\") pod \"1aba2032-5797-4934-ab3b-42c7095d6c0a\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " Oct 31 01:12:42.463142 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.463009 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca\") pod \"1aba2032-5797-4934-ab3b-42c7095d6c0a\" (UID: \"1aba2032-5797-4934-ab3b-42c7095d6c0a\") " Oct 31 01:12:42.463317 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.463298 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca" pod "1aba2032-5797-4934-ab3b-42c7095d6c0a" (UID: "1aba2032-5797-4934-ab3b-42c7095d6c0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:42.465277 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.465250 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca" pod "1aba2032-5797-4934-ab3b-42c7095d6c0a" (UID: "1aba2032-5797-4934-ab3b-42c7095d6c0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:42.465277 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.465274 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx" pod "1aba2032-5797-4934-ab3b-42c7095d6c0a" (UID: "1aba2032-5797-4934-ab3b-42c7095d6c0a"). InnerVolumeSpecName "kube-api-access-bb7lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:42.564009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.563973 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:42.564009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.564001 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aba2032-5797-4934-ab3b-42c7095d6c0a-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:42.564009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:42.564013 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7lx\" (UniqueName: \"kubernetes.io/projected/1aba2032-5797-4934-ab3b-42c7095d6c0a-kube-api-access-bb7lx\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:43.295032 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:43.294988 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" event={"ID":"1aba2032-5797-4934-ab3b-42c7095d6c0a","Type":"ContainerDied","Data":"f5bcfae43368f42e607bff5c3a3ca31dab97d1d4228ef29dbd2abda195813d8c"} Oct 31 01:12:43.295457 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:43.295057 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5bcfae43368f42e607bff5c3a3ca31dab97d1d4228ef29dbd2abda195813d8c" Oct 31 01:12:43.295457 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:43.295060 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d4bcfd-9lq95" Oct 31 01:12:45.824502 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:45.824465 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:47.457560 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:47.457527 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:47.615340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:47.615298 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:47.830328 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:47.830295 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww"] Oct 31 01:12:47.830514 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:47.830498 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" containerID="cri-o://0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53" gracePeriod=30 Oct 31 01:12:47.837828 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:47.837792 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:50.561846 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:50.561743 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:51.620933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:51.620899 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:51.766829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:51.766788 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:52.469254 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.469215 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8"] Oct 31 01:12:52.469679 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.469665 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1aba2032-5797-4934-ab3b-42c7095d6c0a" containerName="revoke-keys" Oct 31 01:12:52.469730 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.469681 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aba2032-5797-4934-ab3b-42c7095d6c0a" containerName="revoke-keys" Oct 31 01:12:52.469827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.469817 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aba2032-5797-4934-ab3b-42c7095d6c0a" containerName="revoke-keys" Oct 31 01:12:52.472912 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.472892 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.525041 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.525001 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:12:52.525226 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.525044 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:12:52.564794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.564749 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.564794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.564785 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.565056 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.564961 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnf6\" (UniqueName: \"kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.574655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.574624 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:12:52.574655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.574624 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:12:52.577042 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.577015 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8"] Oct 31 01:12:52.666045 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.666010 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnf6\" (UniqueName: \"kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.666484 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.666084 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.666484 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.666111 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.666933 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.666914 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.668768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.668735 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.727826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.727757 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnf6\" (UniqueName: \"kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6\") pod \"maas-api-revoke-sub-009adcdb-89pw8\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:52.786039 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:52.786007 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:53.079128 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.079101 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 01:12:53.166378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.166355 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8"] Oct 31 01:12:53.167975 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:12:53.167950 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb397ab0c_73f9_4ef2_a403_14d3ed5fcb05.slice/crio-af61c441c80293befd1ed41bf851f3a94429b25b861c9aaa12ff79626ff2b76a WatchSource:0}: Error finding container af61c441c80293befd1ed41bf851f3a94429b25b861c9aaa12ff79626ff2b76a: Status 404 returned error can't find the container with id af61c441c80293befd1ed41bf851f3a94429b25b861c9aaa12ff79626ff2b76a Oct 31 01:12:53.172812 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.172790 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.172920 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.172904 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.172992 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.172976 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173014 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173108 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173049 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173160 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173104 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173157 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173168 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:53.173239 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173223 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173271 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket\") pod \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\" (UID: \"e346dc38-bb4a-41ec-80a4-5f9fd9478e34\") " Oct 31 01:12:53.173392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173351 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:53.173392 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173377 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:53.173549 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173522 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:53.173658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173632 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:53.173769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173694 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.173769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173714 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.173769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173731 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.173769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.173744 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.175377 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.175351 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:53.175680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.175661 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "kube-api-access-mmzpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:53.175976 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.175951 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:12:53.175976 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.175966 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token" pod "e346dc38-bb4a-41ec-80a4-5f9fd9478e34" (UID: "e346dc38-bb4a-41ec-80a4-5f9fd9478e34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:53.275168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.275093 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.275168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.275117 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.275168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.275128 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.275168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.275137 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmzpf\" (UniqueName: \"kubernetes.io/projected/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-kube-api-access-mmzpf\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.275168 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.275149 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e346dc38-bb4a-41ec-80a4-5f9fd9478e34-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:53.358757 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.358724 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" event={"ID":"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05","Type":"ContainerStarted","Data":"af61c441c80293befd1ed41bf851f3a94429b25b861c9aaa12ff79626ff2b76a"} Oct 31 01:12:53.360697 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.360670 5943 generic.go:356] "Generic (PLEG): container finished" podID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerID="0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53" exitCode=0 Oct 31 01:12:53.360847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.360734 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" event={"ID":"e346dc38-bb4a-41ec-80a4-5f9fd9478e34","Type":"ContainerDied","Data":"0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53"} Oct 31 01:12:53.360847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.360744 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" Oct 31 01:12:53.360847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.360771 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww" event={"ID":"e346dc38-bb4a-41ec-80a4-5f9fd9478e34","Type":"ContainerDied","Data":"ec2dddf6820b96331784e9d308d559a657a0fda16b39b9fbdf0cf007b0d636c5"} Oct 31 01:12:53.360847 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.360792 5943 scope.go:122] "RemoveContainer" containerID="0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53" Oct 31 01:12:53.371255 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.371233 5943 scope.go:122] "RemoveContainer" containerID="28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe" Oct 31 01:12:53.380378 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.380357 5943 scope.go:122] "RemoveContainer" containerID="0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53" Oct 31 01:12:53.380647 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:12:53.380629 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53\": container with ID starting with 0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53 not found: ID does not exist" containerID="0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53" Oct 31 01:12:53.380696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.380657 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53"} err="failed to get container status \"0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53\": rpc error: code = NotFound desc = could not find container \"0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53\": container with ID starting with 0f3f8a83e1f50568bf301fb5edef904ab346a086c1346175040aecb11b691a53 not found: ID does not exist" Oct 31 01:12:53.380696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.380675 5943 scope.go:122] "RemoveContainer" containerID="28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe" Oct 31 01:12:53.380927 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:12:53.380910 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe\": container with ID starting with 28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe not found: ID does not exist" containerID="28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe" Oct 31 01:12:53.380969 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.380934 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe"} err="failed to get container status \"28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe\": rpc error: code = NotFound desc = could not find container \"28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe\": container with ID starting with 28d653169e28c7bb9e7ea811cef01805aaf42f402bf5743214677d4dec7f13fe not found: ID does not exist" Oct 31 01:12:53.525000 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.524961 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww"] Oct 31 01:12:53.534741 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:53.534056 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zzwww"] Oct 31 01:12:54.040862 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:54.040827 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" path="/var/lib/kubelet/pods/e346dc38-bb4a-41ec-80a4-5f9fd9478e34/volumes" Oct 31 01:12:54.368339 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:54.368298 5943 generic.go:356] "Generic (PLEG): container finished" podID="b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" containerID="2b3b1fcb5cabd18f709a52bc97999f413d682b42e329a0f45306af78ca059433" exitCode=0 Oct 31 01:12:54.368555 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:54.368385 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" event={"ID":"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05","Type":"ContainerDied","Data":"2b3b1fcb5cabd18f709a52bc97999f413d682b42e329a0f45306af78ca059433"} Oct 31 01:12:55.343067 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.343034 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:55.353347 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.353320 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:12:55.515258 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.515227 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:55.602120 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.602042 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6\" (UniqueName: \"kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6\") pod \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " Oct 31 01:12:55.602309 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.602167 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca\") pod \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " Oct 31 01:12:55.602309 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.602224 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca\") pod \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\" (UID: \"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05\") " Oct 31 01:12:55.602618 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.602593 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca" pod "b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" (UID: "b397ab0c-73f9-4ef2-a403-14d3ed5fcb05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:55.604280 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.604259 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca" pod "b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" (UID: "b397ab0c-73f9-4ef2-a403-14d3ed5fcb05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:55.604373 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.604354 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6" pod "b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" (UID: "b397ab0c-73f9-4ef2-a403-14d3ed5fcb05"). InnerVolumeSpecName "kube-api-access-5dnf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:55.703143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.703107 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:55.703143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.703136 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:55.703143 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:55.703148 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnf6\" (UniqueName: \"kubernetes.io/projected/b397ab0c-73f9-4ef2-a403-14d3ed5fcb05-kube-api-access-5dnf6\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:56.382164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:56.382084 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" Oct 31 01:12:56.382164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:56.382083 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009adcdb-89pw8" event={"ID":"b397ab0c-73f9-4ef2-a403-14d3ed5fcb05","Type":"ContainerDied","Data":"af61c441c80293befd1ed41bf851f3a94429b25b861c9aaa12ff79626ff2b76a"} Oct 31 01:12:56.382164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:12:56.382138 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af61c441c80293befd1ed41bf851f3a94429b25b861c9aaa12ff79626ff2b76a" Oct 31 01:12:57.475523 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:12:57.475482 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:13:00.062480 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:00.062446 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:02.171743 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:02.171700 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:04.838836 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:04.838797 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:04.987758 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:04.987726 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:05.183196 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:05.183096 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:05.638594 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:05.638554 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:09.682124 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:09.682076 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:11.714948 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:11.714913 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:11.910351 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:11.910314 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:14.442874 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:14.442836 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:17.074570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:17.074527 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:13:17.956465 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:17.956430 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:18.153109 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.153074 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:18.514230 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514198 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g"] Oct 31 01:13:18.514692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514677 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" Oct 31 01:13:18.514692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514693 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" Oct 31 01:13:18.514840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514706 5943 cpu_manager.go:414] "containerMap: removing container" podUID="b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" containerName="revoke-keys" Oct 31 01:13:18.514840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514711 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" containerName="revoke-keys" Oct 31 01:13:18.514840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514801 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" Oct 31 01:13:18.514840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514812 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" Oct 31 01:13:18.514840 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.514818 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="b397ab0c-73f9-4ef2-a403-14d3ed5fcb05" containerName="revoke-keys" Oct 31 01:13:18.518003 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.517986 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.540398 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.540371 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.540519 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.540431 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b54b\" (UniqueName: \"kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.540519 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.540504 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.619788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.619753 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:18.619788 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.619765 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:13:18.620058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.619794 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:13:18.620058 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.619792 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:13:18.622000 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.621980 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g"] Oct 31 01:13:18.641495 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.641466 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.641645 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.641540 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b54b\" (UniqueName: \"kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.641645 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.641606 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.642147 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.642118 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.644107 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.644082 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.702352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.702316 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b54b\" (UniqueName: \"kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b\") pod \"maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:18.831316 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:18.831232 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:19.216872 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:19.206479 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g"] Oct 31 01:13:19.216872 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:13:19.208413 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b89b772_4e1e_479b_9a9b_52c808645aa3.slice/crio-9cef5f173cf5cebfacb9450d89e2fc54bb0f76648ed0e0d92c7a9dc53893fd72 WatchSource:0}: Error finding container 9cef5f173cf5cebfacb9450d89e2fc54bb0f76648ed0e0d92c7a9dc53893fd72: Status 404 returned error can't find the container with id 9cef5f173cf5cebfacb9450d89e2fc54bb0f76648ed0e0d92c7a9dc53893fd72 Oct 31 01:13:19.396619 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:19.396572 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:19.525756 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:19.525704 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" event={"ID":"2b89b772-4e1e-479b-9a9b-52c808645aa3","Type":"ContainerStarted","Data":"9cef5f173cf5cebfacb9450d89e2fc54bb0f76648ed0e0d92c7a9dc53893fd72"} Oct 31 01:13:20.533312 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:20.533274 5943 generic.go:356] "Generic (PLEG): container finished" podID="2b89b772-4e1e-479b-9a9b-52c808645aa3" containerID="d2baeb1407834e9d56061ca36b3c2a6584d313404ac50856b5a178eaf20da708" exitCode=0 Oct 31 01:13:20.533729 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:20.533373 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" event={"ID":"2b89b772-4e1e-479b-9a9b-52c808645aa3","Type":"ContainerDied","Data":"d2baeb1407834e9d56061ca36b3c2a6584d313404ac50856b5a178eaf20da708"} Oct 31 01:13:21.677122 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.677095 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:21.771817 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.771783 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca\") pod \"2b89b772-4e1e-479b-9a9b-52c808645aa3\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " Oct 31 01:13:21.771997 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.771925 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b\" (UniqueName: \"kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b\") pod \"2b89b772-4e1e-479b-9a9b-52c808645aa3\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " Oct 31 01:13:21.772059 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.771994 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca\") pod \"2b89b772-4e1e-479b-9a9b-52c808645aa3\" (UID: \"2b89b772-4e1e-479b-9a9b-52c808645aa3\") " Oct 31 01:13:21.772361 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.772339 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca" pod "2b89b772-4e1e-479b-9a9b-52c808645aa3" (UID: "2b89b772-4e1e-479b-9a9b-52c808645aa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:21.774010 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.773986 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca" pod "2b89b772-4e1e-479b-9a9b-52c808645aa3" (UID: "2b89b772-4e1e-479b-9a9b-52c808645aa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:21.774240 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.774224 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b" pod "2b89b772-4e1e-479b-9a9b-52c808645aa3" (UID: "2b89b772-4e1e-479b-9a9b-52c808645aa3"). InnerVolumeSpecName "kube-api-access-4b54b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:21.873781 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.873690 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:21.873781 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.873722 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b89b772-4e1e-479b-9a9b-52c808645aa3-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:21.873781 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:21.873735 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b54b\" (UniqueName: \"kubernetes.io/projected/2b89b772-4e1e-479b-9a9b-52c808645aa3-kube-api-access-4b54b\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:22.547108 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:22.547077 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" Oct 31 01:13:22.547320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:22.547102 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g" event={"ID":"2b89b772-4e1e-479b-9a9b-52c808645aa3","Type":"ContainerDied","Data":"9cef5f173cf5cebfacb9450d89e2fc54bb0f76648ed0e0d92c7a9dc53893fd72"} Oct 31 01:13:22.547320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:22.547158 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cef5f173cf5cebfacb9450d89e2fc54bb0f76648ed0e0d92c7a9dc53893fd72" Oct 31 01:13:24.485610 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:24.485568 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:27.474670 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:13:27.474623 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:13:29.254028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:29.253995 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:30.099065 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:30.099033 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692"] Oct 31 01:13:30.207746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:30.207713 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn"] Oct 31 01:13:30.464019 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:30.463922 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:30.657807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:30.657769 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:34.052426 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:34.052390 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:35.099608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.099563 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" podUID="02086238-cbd8-4e28-a364-e2de71af6709" containerName="payload-processing" containerID="cri-o://b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4" gracePeriod=25 Oct 31 01:13:35.208978 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.208923 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" podUID="29680087-3751-4d74-8a11-c3364bbef202" containerName="payload-pre-processing" containerID="cri-o://02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07" gracePeriod=25 Oct 31 01:13:35.360832 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.360771 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:13:35.410584 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.410560 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume\") pod \"02086238-cbd8-4e28-a364-e2de71af6709\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " Oct 31 01:13:35.410754 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.410631 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b\" (UniqueName: \"kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b\") pod \"02086238-cbd8-4e28-a364-e2de71af6709\" (UID: \"02086238-cbd8-4e28-a364-e2de71af6709\") " Oct 31 01:13:35.410964 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.410938 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume" pod "02086238-cbd8-4e28-a364-e2de71af6709" (UID: "02086238-cbd8-4e28-a364-e2de71af6709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:35.412842 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.412814 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b" pod "02086238-cbd8-4e28-a364-e2de71af6709" (UID: "02086238-cbd8-4e28-a364-e2de71af6709"). InnerVolumeSpecName "kube-api-access-pqw5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:35.466237 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.466166 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:13:35.512267 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.512233 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqw5b\" (UniqueName: \"kubernetes.io/projected/02086238-cbd8-4e28-a364-e2de71af6709-kube-api-access-pqw5b\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:35.512267 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.512263 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02086238-cbd8-4e28-a364-e2de71af6709-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:35.613203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.613115 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume\") pod \"29680087-3751-4d74-8a11-c3364bbef202\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " Oct 31 01:13:35.613203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.613200 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg\" (UniqueName: \"kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg\") pod \"29680087-3751-4d74-8a11-c3364bbef202\" (UID: \"29680087-3751-4d74-8a11-c3364bbef202\") " Oct 31 01:13:35.613658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.613629 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume" pod "29680087-3751-4d74-8a11-c3364bbef202" (UID: "29680087-3751-4d74-8a11-c3364bbef202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:35.615515 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.615490 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg" pod "29680087-3751-4d74-8a11-c3364bbef202" (UID: "29680087-3751-4d74-8a11-c3364bbef202"). InnerVolumeSpecName "kube-api-access-nh2hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:35.631461 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.631436 5943 generic.go:356] "Generic (PLEG): container finished" podID="29680087-3751-4d74-8a11-c3364bbef202" containerID="02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07" exitCode=0 Oct 31 01:13:35.631563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.631490 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" Oct 31 01:13:35.631563 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.631518 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" event={"ID":"29680087-3751-4d74-8a11-c3364bbef202","Type":"ContainerDied","Data":"02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07"} Oct 31 01:13:35.631652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.631577 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn" event={"ID":"29680087-3751-4d74-8a11-c3364bbef202","Type":"ContainerDied","Data":"c2d3994caf7cab8933445bd9e717ae8ad30e79d7bdd52c98cda78be548c440a7"} Oct 31 01:13:35.631652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.631607 5943 scope.go:122] "RemoveContainer" containerID="02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07" Oct 31 01:13:35.633149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.633125 5943 generic.go:356] "Generic (PLEG): container finished" podID="02086238-cbd8-4e28-a364-e2de71af6709" containerID="b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4" exitCode=0 Oct 31 01:13:35.633293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.633191 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" Oct 31 01:13:35.633293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.633208 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" event={"ID":"02086238-cbd8-4e28-a364-e2de71af6709","Type":"ContainerDied","Data":"b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4"} Oct 31 01:13:35.633293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.633252 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692" event={"ID":"02086238-cbd8-4e28-a364-e2de71af6709","Type":"ContainerDied","Data":"c365dec5fbe9fdd5f29a217bfa79031eaecded86dddbf6dc256cb532ad881396"} Oct 31 01:13:35.642611 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.642585 5943 scope.go:122] "RemoveContainer" containerID="02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07" Oct 31 01:13:35.642899 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:13:35.642877 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07\": container with ID starting with 02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07 not found: ID does not exist" containerID="02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07" Oct 31 01:13:35.642989 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.642908 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07"} err="failed to get container status \"02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07\": rpc error: code = NotFound desc = could not find container \"02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07\": container with ID starting with 02e29e028584db9f8219e92ec2c6e71819cbc573abbcbcf5d46003282f454c07 not found: ID does not exist" Oct 31 01:13:35.642989 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.642930 5943 scope.go:122] "RemoveContainer" containerID="b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4" Oct 31 01:13:35.653142 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.653122 5943 scope.go:122] "RemoveContainer" containerID="b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4" Oct 31 01:13:35.653451 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:13:35.653430 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4\": container with ID starting with b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4 not found: ID does not exist" containerID="b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4" Oct 31 01:13:35.653525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.653457 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4"} err="failed to get container status \"b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4\": rpc error: code = NotFound desc = could not find container \"b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4\": container with ID starting with b0d8a6e5fca7575f7e41cee6a3f7f9952ebbf3354da473b1952cbadad28495d4 not found: ID does not exist" Oct 31 01:13:35.714137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.714105 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh2hg\" (UniqueName: \"kubernetes.io/projected/29680087-3751-4d74-8a11-c3364bbef202-kube-api-access-nh2hg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:35.714137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.714134 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29680087-3751-4d74-8a11-c3364bbef202-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:35.796768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.796733 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn"] Oct 31 01:13:35.800320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.800298 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3853b6-5598d7cf7f-87kpn"] Oct 31 01:13:35.909527 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.909431 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692"] Oct 31 01:13:35.913338 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:35.913306 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3853b6-56ff76557c-qk692"] Oct 31 01:13:36.040438 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:36.040403 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02086238-cbd8-4e28-a364-e2de71af6709" path="/var/lib/kubelet/pods/02086238-cbd8-4e28-a364-e2de71af6709/volumes" Oct 31 01:13:36.040781 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:36.040767 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29680087-3751-4d74-8a11-c3364bbef202" path="/var/lib/kubelet/pods/29680087-3751-4d74-8a11-c3364bbef202/volumes" Oct 31 01:13:37.306909 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:37.306874 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:38.245050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:38.245014 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:39.744113 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:39.744074 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:40.020240 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:40.020210 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:40.164111 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:40.164074 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:40.237717 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:40.237684 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g"] Oct 31 01:13:40.240320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:40.240299 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3853b6-j268g"] Oct 31 01:13:40.699270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:40.699232 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:42.041340 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:42.041298 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b89b772-4e1e-479b-9a9b-52c808645aa3" path="/var/lib/kubelet/pods/2b89b772-4e1e-479b-9a9b-52c808645aa3/volumes" Oct 31 01:13:42.205641 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:42.205608 5943 scope.go:122] "RemoveContainer" containerID="a307b07f074e576d3eb0f986fd4183e7d1a0e3f863021acafef5d1ff8661c6ad" Oct 31 01:13:42.215052 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:42.215027 5943 scope.go:122] "RemoveContainer" containerID="e55e6c97630e6a5f9a9ace2d1cae8a39dee78966dd6ebb18a5d4f3ae30123019" Oct 31 01:13:42.224667 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:42.224648 5943 scope.go:122] "RemoveContainer" containerID="20f563fe2ff5fdfb71d6364999130c4eff9c4b4210b750a66f299086332f858f" Oct 31 01:13:44.243549 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:44.243517 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:13:45.157081 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:45.157044 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:46.539680 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:46.539643 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:46.690069 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:46.690026 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:46.894395 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:46.894309 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h"] Oct 31 01:13:46.894562 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:46.894519 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" containerID="cri-o://cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec" gracePeriod=30 Oct 31 01:13:46.911919 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:46.911886 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:47.822920 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.822890 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x"] Oct 31 01:13:47.823435 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823419 5943 cpu_manager.go:414] "containerMap: removing container" podUID="29680087-3751-4d74-8a11-c3364bbef202" containerName="payload-pre-processing" Oct 31 01:13:47.823483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823438 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29680087-3751-4d74-8a11-c3364bbef202" containerName="payload-pre-processing" Oct 31 01:13:47.823483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823452 5943 cpu_manager.go:414] "containerMap: removing container" podUID="02086238-cbd8-4e28-a364-e2de71af6709" containerName="payload-processing" Oct 31 01:13:47.823483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823459 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02086238-cbd8-4e28-a364-e2de71af6709" containerName="payload-processing" Oct 31 01:13:47.823585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823489 5943 cpu_manager.go:414] "containerMap: removing container" podUID="2b89b772-4e1e-479b-9a9b-52c808645aa3" containerName="revoke-keys" Oct 31 01:13:47.823585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823494 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b89b772-4e1e-479b-9a9b-52c808645aa3" containerName="revoke-keys" Oct 31 01:13:47.823585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823505 5943 cpu_manager.go:414] "containerMap: removing container" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" Oct 31 01:13:47.823585 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823510 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e346dc38-bb4a-41ec-80a4-5f9fd9478e34" containerName="istio-proxy" Oct 31 01:13:47.823707 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823591 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="29680087-3751-4d74-8a11-c3364bbef202" containerName="payload-pre-processing" Oct 31 01:13:47.823707 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823599 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="02086238-cbd8-4e28-a364-e2de71af6709" containerName="payload-processing" Oct 31 01:13:47.823707 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.823609 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b89b772-4e1e-479b-9a9b-52c808645aa3" containerName="revoke-keys" Oct 31 01:13:47.826872 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.826858 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:47.878811 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.878784 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:47.878998 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.878856 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:13:47.928493 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.928459 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:13:47.928493 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.928460 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:13:47.930567 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.930547 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x"] Oct 31 01:13:47.938984 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.938956 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:47.939098 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.939023 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvs95\" (UniqueName: \"kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:47.939156 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:47.939100 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.040471 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.040442 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.040655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.040505 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvs95\" (UniqueName: \"kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.040655 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.040528 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.041209 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.041156 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.043105 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.043084 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.102365 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.102288 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvs95\" (UniqueName: \"kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95\") pod \"maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.139977 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.139936 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:48.512554 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.512390 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x"] Oct 31 01:13:48.515008 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:13:48.514978 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d48897b_1013_4978_9114_45645f78caed.slice/crio-60fbb12fcfd3d6b7ff77b4bc43ed988047407d4c663761f289cc5fc258bf093e WatchSource:0}: Error finding container 60fbb12fcfd3d6b7ff77b4bc43ed988047407d4c663761f289cc5fc258bf093e: Status 404 returned error can't find the container with id 60fbb12fcfd3d6b7ff77b4bc43ed988047407d4c663761f289cc5fc258bf093e Oct 31 01:13:48.723668 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:48.723634 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" event={"ID":"9d48897b-1013-4978-9114-45645f78caed","Type":"ContainerStarted","Data":"60fbb12fcfd3d6b7ff77b4bc43ed988047407d4c663761f289cc5fc258bf093e"} Oct 31 01:13:49.730848 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:49.730807 5943 generic.go:356] "Generic (PLEG): container finished" podID="9d48897b-1013-4978-9114-45645f78caed" containerID="f706ec8e2ec473ea354afefa6e7c232cdad5a33d32dc09180409ba912de961fa" exitCode=0 Oct 31 01:13:49.731316 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:49.730901 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" event={"ID":"9d48897b-1013-4978-9114-45645f78caed","Type":"ContainerDied","Data":"f706ec8e2ec473ea354afefa6e7c232cdad5a33d32dc09180409ba912de961fa"} Oct 31 01:13:50.154652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.154617 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.871875 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.871852 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:50.969959 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.969927 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca\") pod \"9d48897b-1013-4978-9114-45645f78caed\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " Oct 31 01:13:50.970164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.970042 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca\") pod \"9d48897b-1013-4978-9114-45645f78caed\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " Oct 31 01:13:50.970164 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.970122 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95\" (UniqueName: \"kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95\") pod \"9d48897b-1013-4978-9114-45645f78caed\" (UID: \"9d48897b-1013-4978-9114-45645f78caed\") " Oct 31 01:13:50.970490 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.970462 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca" pod "9d48897b-1013-4978-9114-45645f78caed" (UID: "9d48897b-1013-4978-9114-45645f78caed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:50.970611 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.970535 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48897b-1013-4978-9114-45645f78caed-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:50.972244 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.972218 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca" pod "9d48897b-1013-4978-9114-45645f78caed" (UID: "9d48897b-1013-4978-9114-45645f78caed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:50.972244 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:50.972222 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95" pod "9d48897b-1013-4978-9114-45645f78caed" (UID: "9d48897b-1013-4978-9114-45645f78caed"). InnerVolumeSpecName "kube-api-access-tvs95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:51.071247 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:51.071210 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvs95\" (UniqueName: \"kubernetes.io/projected/9d48897b-1013-4978-9114-45645f78caed-kube-api-access-tvs95\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:51.071247 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:51.071241 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48897b-1013-4978-9114-45645f78caed-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:51.743660 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:51.743630 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" Oct 31 01:13:51.743660 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:51.743653 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x" event={"ID":"9d48897b-1013-4978-9114-45645f78caed","Type":"ContainerDied","Data":"60fbb12fcfd3d6b7ff77b4bc43ed988047407d4c663761f289cc5fc258bf093e"} Oct 31 01:13:51.743926 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:51.743697 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fbb12fcfd3d6b7ff77b4bc43ed988047407d4c663761f289cc5fc258bf093e" Oct 31 01:13:52.155974 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.155941 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:13:52.285849 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.285766 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.285849 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.285841 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286092 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.285931 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286092 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.285970 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286092 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286013 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286092 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286043 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286127 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286158 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286240 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data\") pod \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\" (UID: \"ecf5c579-b3d6-429b-b91b-b45c109dfb09\") " Oct 31 01:13:52.286363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286262 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:52.286579 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286447 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:52.286579 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286460 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:52.286858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286780 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.287159 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.287135 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.287294 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.287162 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.287294 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.286822 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:52.287294 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.287263 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:52.288433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.288404 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "kube-api-access-79g24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:52.288558 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.288514 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:52.288661 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.288643 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:52.288912 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.288894 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo" pod "ecf5c579-b3d6-429b-b91b-b45c109dfb09" (UID: "ecf5c579-b3d6-429b-b91b-b45c109dfb09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:13:52.388441 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.388404 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79g24\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-kube-api-access-79g24\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.388441 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.388435 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.388441 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.388446 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.388701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.388454 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.388701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.388463 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.388701 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.388473 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf5c579-b3d6-429b-b91b-b45c109dfb09-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:52.752169 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.752076 5943 generic.go:356] "Generic (PLEG): container finished" podID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerID="cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec" exitCode=0 Oct 31 01:13:52.752169 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.752138 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" Oct 31 01:13:52.752169 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.752158 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" event={"ID":"ecf5c579-b3d6-429b-b91b-b45c109dfb09","Type":"ContainerDied","Data":"cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec"} Oct 31 01:13:52.752459 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.752240 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h" event={"ID":"ecf5c579-b3d6-429b-b91b-b45c109dfb09","Type":"ContainerDied","Data":"43365d08a1fb4e389b2d0af2ee4000902dbbe3180a2269da72af61c91ed45602"} Oct 31 01:13:52.752459 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.752264 5943 scope.go:122] "RemoveContainer" containerID="cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec" Oct 31 01:13:52.763007 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.762985 5943 scope.go:122] "RemoveContainer" containerID="b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f" Oct 31 01:13:52.772206 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.772189 5943 scope.go:122] "RemoveContainer" containerID="cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec" Oct 31 01:13:52.772496 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:13:52.772476 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec\": container with ID starting with cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec not found: ID does not exist" containerID="cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec" Oct 31 01:13:52.772553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.772505 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec"} err="failed to get container status \"cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec\": rpc error: code = NotFound desc = could not find container \"cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec\": container with ID starting with cb76643db7ce1ac7eb9f1b9aaa3879d55ac0ff71328de9784f1562415f4fc4ec not found: ID does not exist" Oct 31 01:13:52.772553 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.772525 5943 scope.go:122] "RemoveContainer" containerID="b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f" Oct 31 01:13:52.772774 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:13:52.772754 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f\": container with ID starting with b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f not found: ID does not exist" containerID="b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f" Oct 31 01:13:52.772821 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.772780 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f"} err="failed to get container status \"b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f\": rpc error: code = NotFound desc = could not find container \"b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f\": container with ID starting with b9b682337efedfabd5402745f1e55470200dd8aff6503850491b144cb506437f not found: ID does not exist" Oct 31 01:13:52.916198 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.916147 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h"] Oct 31 01:13:52.954632 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:52.954597 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3853b6-openshift-default-758b695749-tgg5h"] Oct 31 01:13:54.040608 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:54.040573 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" path="/var/lib/kubelet/pods/ecf5c579-b3d6-429b-b91b-b45c109dfb09/volumes" Oct 31 01:13:54.955315 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:54.955270 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.475320 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:13:57.475286 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:13:59.698480 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:59.698441 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:59.893533 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:13:59.893498 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:02.546536 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:02.546487 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:04.647548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:04.647505 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.794633 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:08.794592 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.322635 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:10.322594 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.566366 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:10.566331 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.766744 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:10.766711 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.880067 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:10.880037 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x"] Oct 31 01:14:10.929242 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:10.929207 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d7412d-pbr2x"] Oct 31 01:14:11.335272 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:11.335236 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:12.039981 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:12.039949 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d48897b-1013-4978-9114-45645f78caed" path="/var/lib/kubelet/pods/9d48897b-1013-4978-9114-45645f78caed/volumes" Oct 31 01:14:13.896681 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:13.896647 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:14.587009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:14.586968 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:14:15.647229 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:15.647189 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:17.574026 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:17.573991 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:17.722081 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:17.722049 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:17.965887 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:17.965788 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:18.843011 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.842925 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw"] Oct 31 01:14:18.843625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843605 5943 cpu_manager.go:414] "containerMap: removing container" podUID="9d48897b-1013-4978-9114-45645f78caed" containerName="revoke-keys" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843631 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d48897b-1013-4978-9114-45645f78caed" containerName="revoke-keys" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843672 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843679 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843696 5943 cpu_manager.go:414] "containerMap: removing container" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843704 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843837 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d48897b-1013-4978-9114-45645f78caed" containerName="revoke-keys" Oct 31 01:14:18.843858 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.843853 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" Oct 31 01:14:18.847757 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.847735 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:18.902420 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.902390 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:18.949766 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.949734 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:18.949953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.949794 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngxp\" (UniqueName: \"kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:18.950001 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.949944 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:18.951692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.951669 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:14:18.951829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.951723 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:14:18.951829 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.951809 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:14:18.953808 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:18.953788 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw"] Oct 31 01:14:19.051051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.051014 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.051258 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.051085 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngxp\" (UniqueName: \"kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.051258 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.051240 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.051878 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.051849 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.053597 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.053581 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.112337 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.112260 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngxp\" (UniqueName: \"kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.161417 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.161387 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:19.532962 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.532934 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw"] Oct 31 01:14:19.540705 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:14:19.540679 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f61929d_80aa_4a8e_ba8c_e32aa9a6207a.slice/crio-22518e4d42b02f3ce60783859e8745f06cf6c5fd2581b45ec75f8b5c9ee509e9 WatchSource:0}: Error finding container 22518e4d42b02f3ce60783859e8745f06cf6c5fd2581b45ec75f8b5c9ee509e9: Status 404 returned error can't find the container with id 22518e4d42b02f3ce60783859e8745f06cf6c5fd2581b45ec75f8b5c9ee509e9 Oct 31 01:14:19.916740 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.916697 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" event={"ID":"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a","Type":"ContainerStarted","Data":"4a816f66b957b7cfd222af31bb70e22f4463a9976781da9a5b8aee25293fa82d"} Oct 31 01:14:19.917256 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:19.916782 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" event={"ID":"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a","Type":"ContainerStarted","Data":"22518e4d42b02f3ce60783859e8745f06cf6c5fd2581b45ec75f8b5c9ee509e9"} Oct 31 01:14:20.034002 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:20.033949 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" podStartSLOduration=2.033934021 podStartE2EDuration="2.033934021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:20.032068871 +0000 UTC m=+4240.125359090" watchObservedRunningTime="2026-10-31 01:14:20.033934021 +0000 UTC m=+4240.127224208" Oct 31 01:14:20.691265 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:20.691141 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.924662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:20.924625 5943 generic.go:356] "Generic (PLEG): container finished" podID="1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" containerID="4a816f66b957b7cfd222af31bb70e22f4463a9976781da9a5b8aee25293fa82d" exitCode=0 Oct 31 01:14:20.925083 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:20.924716 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" event={"ID":"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a","Type":"ContainerDied","Data":"4a816f66b957b7cfd222af31bb70e22f4463a9976781da9a5b8aee25293fa82d"} Oct 31 01:14:22.069436 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.069408 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:22.187830 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.187740 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca\") pod \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " Oct 31 01:14:22.187830 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.187796 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca\") pod \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " Oct 31 01:14:22.188044 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.187936 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp\" (UniqueName: \"kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp\") pod \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\" (UID: \"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a\") " Oct 31 01:14:22.188572 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.188538 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca" pod "1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" (UID: "1f61929d-80aa-4a8e-ba8c-e32aa9a6207a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:22.190687 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.190661 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp" pod "1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" (UID: "1f61929d-80aa-4a8e-ba8c-e32aa9a6207a"). InnerVolumeSpecName "kube-api-access-5ngxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:22.190982 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.190960 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca" pod "1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" (UID: "1f61929d-80aa-4a8e-ba8c-e32aa9a6207a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:22.289108 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.289067 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ngxp\" (UniqueName: \"kubernetes.io/projected/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-kube-api-access-5ngxp\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:22.289108 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.289105 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:22.289418 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.289119 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:22.938511 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.938479 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" Oct 31 01:14:22.938688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.938479 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw" event={"ID":"1f61929d-80aa-4a8e-ba8c-e32aa9a6207a","Type":"ContainerDied","Data":"22518e4d42b02f3ce60783859e8745f06cf6c5fd2581b45ec75f8b5c9ee509e9"} Oct 31 01:14:22.938688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:22.938606 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22518e4d42b02f3ce60783859e8745f06cf6c5fd2581b45ec75f8b5c9ee509e9" Oct 31 01:14:25.055940 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:25.055907 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd"] Oct 31 01:14:25.105141 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:25.105103 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5afc23b4-mlcwd"] Oct 31 01:14:25.680706 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:25.680672 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:26.040086 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:26.040054 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f70a13-1039-4ee3-9bb3-26ff9dea2af0" path="/var/lib/kubelet/pods/60f70a13-1039-4ee3-9bb3-26ff9dea2af0/volumes" Oct 31 01:14:26.045986 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:26.045957 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb"] Oct 31 01:14:26.095451 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:26.095421 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4740656-7hkkb"] Oct 31 01:14:27.474895 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:14:27.474858 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:14:28.040293 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:28.040255 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75d3154-1a7b-437e-99dc-92f967abab44" path="/var/lib/kubelet/pods/c75d3154-1a7b-437e-99dc-92f967abab44/volumes" Oct 31 01:14:30.460760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:30.460727 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:30.691714 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:30.691664 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:30.839440 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:30.839398 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:33.718394 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:33.718360 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:33.864457 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:33.864409 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:35.251441 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:35.251401 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:36.049155 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:36.049110 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr"] Oct 31 01:14:36.098600 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:36.098571 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769324f-9mbtr"] Oct 31 01:14:36.100323 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:36.100297 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp"] Oct 31 01:14:36.101652 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:36.101632 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e682311-bd7sp"] Oct 31 01:14:38.040270 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:38.040231 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f725819-eab9-4b01-b31f-84f2812a9428" path="/var/lib/kubelet/pods/3f725819-eab9-4b01-b31f-84f2812a9428/volumes" Oct 31 01:14:38.040668 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:38.040634 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9820980-ef20-441e-8a6e-fed2c8229744" path="/var/lib/kubelet/pods/b9820980-ef20-441e-8a6e-fed2c8229744/volumes" Oct 31 01:14:40.049572 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:40.049538 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:42.271953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:42.271922 5943 scope.go:122] "RemoveContainer" containerID="bc0a288945def0feabc9ac8424c5b166277e3f23d5969207ca908a66418f1a52" Oct 31 01:14:42.281696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:42.281665 5943 scope.go:122] "RemoveContainer" containerID="9569ae42f595475964c39a562482d9e8e6cfbebd3833c2309c53e8f190996513" Oct 31 01:14:42.290422 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:42.290403 5943 scope.go:122] "RemoveContainer" containerID="dd807bcbea8e95d244ffdd6044d37a0f90f7d568e9fde11989afe90a012c0f59" Oct 31 01:14:42.298956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:42.298935 5943 scope.go:122] "RemoveContainer" containerID="73f5e2811b132fc26da70ab4b95f2e42ae1dff793094214dad7a041e5c180507" Oct 31 01:14:44.399146 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:44.399107 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:45.837173 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:45.837121 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.054994 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:46.054953 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.197688 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:46.197593 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.380471 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:46.380432 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw"] Oct 31 01:14:46.429794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:46.429757 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4743f8-kk6hw"] Oct 31 01:14:46.782697 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:46.782662 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:48.040960 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:48.040921 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" path="/var/lib/kubelet/pods/1f61929d-80aa-4a8e-ba8c-e32aa9a6207a/volumes" Oct 31 01:14:50.105292 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:50.105258 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:14:51.048375 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:51.048342 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:52.756916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:52.756873 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:52.903971 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:52.903925 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:53.125649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:53.125573 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:56.055088 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:14:56.055049 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:57.475815 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:14:57.475773 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:15:00.162491 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.162463 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx"] Oct 31 01:15:00.163101 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.163081 5943 cpu_manager.go:414] "containerMap: removing container" podUID="1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" containerName="revoke-keys" Oct 31 01:15:00.163216 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.163104 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" containerName="revoke-keys" Oct 31 01:15:00.163287 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.163232 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f61929d-80aa-4a8e-ba8c-e32aa9a6207a" containerName="revoke-keys" Oct 31 01:15:00.163287 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.163257 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf5c579-b3d6-429b-b91b-b45c109dfb09" containerName="istio-proxy" Oct 31 01:15:00.166662 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.166642 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.219005 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.218971 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:15:00.262923 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.262885 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwlm\" (UniqueName: \"kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.263125 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.263069 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.268543 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.268513 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:15:00.269460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.268543 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:15:00.271453 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.271423 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx"] Oct 31 01:15:00.364709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.364677 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.364920 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.364816 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwlm\" (UniqueName: \"kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.365388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.365365 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.425851 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.425770 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwlm\" (UniqueName: \"kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.479734 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.479690 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:00.854952 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.854925 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx"] Oct 31 01:15:00.857749 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:15:00.857712 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4ad16f_1be6_4cb1_a9a6_e02ce871e6fc.slice/crio-63c664cb7ec626bce91feb86bea210d67102e6fbf599afe5ff77f1e0b1bd9154 WatchSource:0}: Error finding container 63c664cb7ec626bce91feb86bea210d67102e6fbf599afe5ff77f1e0b1bd9154: Status 404 returned error can't find the container with id 63c664cb7ec626bce91feb86bea210d67102e6fbf599afe5ff77f1e0b1bd9154 Oct 31 01:15:00.956715 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:00.956676 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:01.161550 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:01.161514 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" event={"ID":"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc","Type":"ContainerStarted","Data":"576b36e94d7eca40c28d8f2dd8b141cdc4e96a473e27199974e8e1c2e21f8956"} Oct 31 01:15:01.161723 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:01.161572 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" event={"ID":"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc","Type":"ContainerStarted","Data":"63c664cb7ec626bce91feb86bea210d67102e6fbf599afe5ff77f1e0b1bd9154"} Oct 31 01:15:01.277491 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:01.277423 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" podStartSLOduration=1.27740623 podStartE2EDuration="1.27740623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.275052025 +0000 UTC m=+4281.368342234" watchObservedRunningTime="2026-10-31 01:15:01.27740623 +0000 UTC m=+4281.370696416" Oct 31 01:15:02.168535 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:02.168499 5943 generic.go:356] "Generic (PLEG): container finished" podID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerID="576b36e94d7eca40c28d8f2dd8b141cdc4e96a473e27199974e8e1c2e21f8956" exitCode=60 Oct 31 01:15:02.168807 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:02.168542 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" event={"ID":"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc","Type":"ContainerDied","Data":"576b36e94d7eca40c28d8f2dd8b141cdc4e96a473e27199974e8e1c2e21f8956"} Oct 31 01:15:02.168856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:02.168812 5943 scope.go:122] "RemoveContainer" containerID="576b36e94d7eca40c28d8f2dd8b141cdc4e96a473e27199974e8e1c2e21f8956" Oct 31 01:15:02.902028 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:02.901993 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:03.176609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:03.176521 5943 generic.go:356] "Generic (PLEG): container finished" podID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerID="69c4e9e133ad32911f07b062379e03d536843335cfb0b2932f261f45cdb11d8b" exitCode=60 Oct 31 01:15:03.176772 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:03.176600 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" event={"ID":"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc","Type":"ContainerDied","Data":"69c4e9e133ad32911f07b062379e03d536843335cfb0b2932f261f45cdb11d8b"} Oct 31 01:15:03.176772 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:03.176659 5943 scope.go:122] "RemoveContainer" containerID="576b36e94d7eca40c28d8f2dd8b141cdc4e96a473e27199974e8e1c2e21f8956" Oct 31 01:15:03.176921 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:03.176905 5943 scope.go:122] "RemoveContainer" containerID="69c4e9e133ad32911f07b062379e03d536843335cfb0b2932f261f45cdb11d8b" Oct 31 01:15:03.177204 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:03.177164 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx_odh-ai-gateway-infra(7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" Oct 31 01:15:04.184456 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:04.184429 5943 scope.go:122] "RemoveContainer" containerID="69c4e9e133ad32911f07b062379e03d536843335cfb0b2932f261f45cdb11d8b" Oct 31 01:15:04.184858 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:04.184666 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx_odh-ai-gateway-infra(7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" Oct 31 01:15:05.047066 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.047032 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh"] Oct 31 01:15:05.098572 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.098517 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5bdfdb-sbmwh"] Oct 31 01:15:05.100980 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.100958 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r"] Oct 31 01:15:05.103203 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.103159 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e40865f-7r78r"] Oct 31 01:15:05.543943 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.543909 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf"] Oct 31 01:15:05.750578 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.750541 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:05.905241 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:05.905126 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:06.040596 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:06.040562 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82bb5cda-962d-4b64-855e-29bc388a375e" path="/var/lib/kubelet/pods/82bb5cda-962d-4b64-855e-29bc388a375e/volumes" Oct 31 01:15:06.041014 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:06.040999 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2eed7b-85fa-4762-a7a5-3024f050d52e" path="/var/lib/kubelet/pods/cf2eed7b-85fa-4762-a7a5-3024f050d52e/volumes" Oct 31 01:15:06.047843 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:06.047813 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:10.544460 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:10.544413 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerName="payload-processing" containerID="cri-o://09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58" gracePeriod=25 Oct 31 01:15:10.558397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:10.558354 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:13.972822 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:13.972781 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:14.793352 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:14.793311 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:16.034908 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:16.034878 5943 scope.go:122] "RemoveContainer" containerID="69c4e9e133ad32911f07b062379e03d536843335cfb0b2932f261f45cdb11d8b" Oct 31 01:15:16.281075 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:16.281044 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:16.553623 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:16.553573 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:16.697540 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:16.697429 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:17.245150 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:17.245104 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:17.257932 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:17.257903 5943 generic.go:356] "Generic (PLEG): container finished" podID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerID="e98b05f492d13ab39b4e463dd2ff3e66b22a6270ade4e5fc3b356e1a70f67995" exitCode=60 Oct 31 01:15:17.258129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:17.257974 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" event={"ID":"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc","Type":"ContainerDied","Data":"e98b05f492d13ab39b4e463dd2ff3e66b22a6270ade4e5fc3b356e1a70f67995"} Oct 31 01:15:17.258129 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:17.258034 5943 scope.go:122] "RemoveContainer" containerID="69c4e9e133ad32911f07b062379e03d536843335cfb0b2932f261f45cdb11d8b" Oct 31 01:15:17.258483 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:17.258465 5943 scope.go:122] "RemoveContainer" containerID="e98b05f492d13ab39b4e463dd2ff3e66b22a6270ade4e5fc3b356e1a70f67995" Oct 31 01:15:17.258765 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:17.258745 5943 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx_odh-ai-gateway-infra(7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" Oct 31 01:15:18.386520 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.386483 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx"] Oct 31 01:15:18.526683 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.526656 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:18.539751 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.539725 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm\" (UniqueName: \"kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm\") pod \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " Oct 31 01:15:18.539891 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.539799 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca\") pod \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\" (UID: \"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc\") " Oct 31 01:15:18.540174 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.540148 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca" pod "7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" (UID: "7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:18.542576 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.542541 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm" pod "7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" (UID: "7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc"). InnerVolumeSpecName "kube-api-access-jkwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.640874 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.640806 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkwlm\" (UniqueName: \"kubernetes.io/projected/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-kube-api-access-jkwlm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.640874 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:18.640830 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc-maas-api-service-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.272400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:19.272368 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" Oct 31 01:15:19.272600 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:19.272397 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx" event={"ID":"7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc","Type":"ContainerDied","Data":"63c664cb7ec626bce91feb86bea210d67102e6fbf599afe5ff77f1e0b1bd9154"} Oct 31 01:15:19.272600 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:19.272462 5943 scope.go:122] "RemoveContainer" containerID="e98b05f492d13ab39b4e463dd2ff3e66b22a6270ade4e5fc3b356e1a70f67995" Oct 31 01:15:19.435999 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:19.435967 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx"] Oct 31 01:15:19.478570 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:19.478529 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890155-2r2zx"] Oct 31 01:15:20.040382 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:20.040352 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" path="/var/lib/kubelet/pods/7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc/volumes" Oct 31 01:15:20.361449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:20.361368 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:15:21.665050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:21.665001 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:23.383397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:23.383346 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:23.532249 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:23.532220 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:23.730522 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:23.730434 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h"] Oct 31 01:15:23.730683 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:23.730663 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" containerID="cri-o://127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68" gracePeriod=30 Oct 31 01:15:23.753007 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:23.752976 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:23.973030 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:23.972990 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:24.233990 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:24.233948 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:26.571760 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:26.571711 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:27.478350 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:27.478311 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:15:28.995796 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:28.995733 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 01:15:29.045855 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.045816 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.045865 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.045888 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.045936 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046033 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.045959 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046047 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046129 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046275 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046205 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046305 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy\") pod \"62b91f93-ae8e-418c-ac6f-53a444084621\" (UID: \"62b91f93-ae8e-418c-ac6f-53a444084621\") " Oct 31 01:15:29.046450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046218 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:29.046450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046335 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:29.046450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046327 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:29.046450 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046368 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:29.046750 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046551 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:29.046881 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046858 5943 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-data\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.047114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046884 5943 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b91f93-ae8e-418c-ac6f-53a444084621-istiod-ca-cert\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.047114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046900 5943 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.047114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046914 5943 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-workload-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.047114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.046929 5943 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-credential-socket\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.048428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.048395 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "kube-api-access-6h2f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:29.048548 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.048523 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:29.048609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.048551 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:29.048609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.048592 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token" pod "62b91f93-ae8e-418c-ac6f-53a444084621" (UID: "62b91f93-ae8e-418c-ac6f-53a444084621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:29.147507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.147471 5943 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b91f93-ae8e-418c-ac6f-53a444084621-istio-podinfo\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.147507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.147499 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2f8\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-kube-api-access-6h2f8\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.147507 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.147510 5943 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b91f93-ae8e-418c-ac6f-53a444084621-istio-token\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.147768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.147520 5943 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b91f93-ae8e-418c-ac6f-53a444084621-istio-envoy\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.216707 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.216673 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf"] Oct 31 01:15:29.218434 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.218402 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg"] Oct 31 01:15:29.294575 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.294543 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:29.331665 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.331633 5943 generic.go:356] "Generic (PLEG): container finished" podID="62b91f93-ae8e-418c-ac6f-53a444084621" containerID="127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68" exitCode=0 Oct 31 01:15:29.332614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.331693 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" Oct 31 01:15:29.332614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.331725 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" event={"ID":"62b91f93-ae8e-418c-ac6f-53a444084621","Type":"ContainerDied","Data":"127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68"} Oct 31 01:15:29.332614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.331793 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h" event={"ID":"62b91f93-ae8e-418c-ac6f-53a444084621","Type":"ContainerDied","Data":"2ea6a482f21a7999efde8b45d228c66fb8e843ec91bea2b728d54473a397d2f4"} Oct 31 01:15:29.332614 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.331831 5943 scope.go:122] "RemoveContainer" containerID="127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68" Oct 31 01:15:29.343207 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.343189 5943 scope.go:122] "RemoveContainer" containerID="4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb" Oct 31 01:15:29.352706 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.352689 5943 scope.go:122] "RemoveContainer" containerID="127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68" Oct 31 01:15:29.352989 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:29.352971 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68\": container with ID starting with 127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68 not found: ID does not exist" containerID="127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68" Oct 31 01:15:29.353040 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.353002 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68"} err="failed to get container status \"127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68\": rpc error: code = NotFound desc = could not find container \"127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68\": container with ID starting with 127c1edc1787796e589d884af56c96e51e95b8fa87f137ff05da8cc8178dbc68 not found: ID does not exist" Oct 31 01:15:29.353040 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.353021 5943 scope.go:122] "RemoveContainer" containerID="4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb" Oct 31 01:15:29.353284 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:29.353268 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb\": container with ID starting with 4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb not found: ID does not exist" containerID="4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb" Oct 31 01:15:29.353341 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.353288 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb"} err="failed to get container status \"4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb\": rpc error: code = NotFound desc = could not find container \"4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb\": container with ID starting with 4428a49e9b0d4ab2574da5ac5cba6289d91986ac4e0e647b7e5b0378dcd76adb not found: ID does not exist" Oct 31 01:15:29.612724 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.612640 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h"] Oct 31 01:15:29.616861 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:29.616834 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-093220-openshift-default-76b9db88df-tnk9h"] Oct 31 01:15:30.040051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.040017 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" path="/var/lib/kubelet/pods/62b91f93-ae8e-418c-ac6f-53a444084621/volumes" Oct 31 01:15:30.798493 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.798468 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 01:15:30.863953 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.863870 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume\") pod \"419e6b2d-8266-46d8-8589-7daa5d359bf6\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " Oct 31 01:15:30.864144 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.863972 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6\" (UniqueName: \"kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6\") pod \"419e6b2d-8266-46d8-8589-7daa5d359bf6\" (UID: \"419e6b2d-8266-46d8-8589-7daa5d359bf6\") " Oct 31 01:15:30.864367 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.864331 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume" pod "419e6b2d-8266-46d8-8589-7daa5d359bf6" (UID: "419e6b2d-8266-46d8-8589-7daa5d359bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:30.866313 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.866286 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6" pod "419e6b2d-8266-46d8-8589-7daa5d359bf6" (UID: "419e6b2d-8266-46d8-8589-7daa5d359bf6"). InnerVolumeSpecName "kube-api-access-7lsw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:30.965746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.965709 5943 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419e6b2d-8266-46d8-8589-7daa5d359bf6-plugins-config-volume\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:30.965746 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:30.965741 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lsw6\" (UniqueName: \"kubernetes.io/projected/419e6b2d-8266-46d8-8589-7daa5d359bf6-kube-api-access-7lsw6\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:31.348324 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.348287 5943 generic.go:356] "Generic (PLEG): container finished" podID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerID="09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58" exitCode=0 Oct 31 01:15:31.348810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.348339 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" Oct 31 01:15:31.348810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.348351 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" event={"ID":"419e6b2d-8266-46d8-8589-7daa5d359bf6","Type":"ContainerDied","Data":"09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58"} Oct 31 01:15:31.348810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.348404 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf" event={"ID":"419e6b2d-8266-46d8-8589-7daa5d359bf6","Type":"ContainerDied","Data":"14b3529e54375f1009342c25d83c253c731f1d55fde617c7221c3a8676e05f3e"} Oct 31 01:15:31.348810 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.348425 5943 scope.go:122] "RemoveContainer" containerID="09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58" Oct 31 01:15:31.358388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.358372 5943 scope.go:122] "RemoveContainer" containerID="09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58" Oct 31 01:15:31.358684 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:31.358664 5943 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58\": container with ID starting with 09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58 not found: ID does not exist" containerID="09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58" Oct 31 01:15:31.358769 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.358695 5943 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58"} err="failed to get container status \"09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58\": rpc error: code = NotFound desc = could not find container \"09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58\": container with ID starting with 09cb8c04444248f8a59d47e0e860d2f5937022a9f6bd3fcc3018c5882d2cad58 not found: ID does not exist" Oct 31 01:15:31.512051 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.512015 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf"] Oct 31 01:15:31.543716 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.543681 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-093220-5479b7bbd7-m6szf"] Oct 31 01:15:31.561005 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:31.560972 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:32.039702 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:32.039672 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" path="/var/lib/kubelet/pods/419e6b2d-8266-46d8-8589-7daa5d359bf6/volumes" Oct 31 01:15:35.811412 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811376 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl"] Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811792 5943 cpu_manager.go:414] "containerMap: removing container" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811803 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811813 5943 cpu_manager.go:414] "containerMap: removing container" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811817 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811834 5943 cpu_manager.go:414] "containerMap: removing container" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811839 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811845 5943 cpu_manager.go:414] "containerMap: removing container" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerName="payload-processing" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811849 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerName="payload-processing" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811856 5943 cpu_manager.go:414] "containerMap: removing container" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" Oct 31 01:15:35.811888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811861 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" Oct 31 01:15:35.812260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811929 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" Oct 31 01:15:35.812260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811940 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.812260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811949 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.812260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811954 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:15:35.812260 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.811962 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="419e6b2d-8266-46d8-8589-7daa5d359bf6" containerName="payload-processing" Oct 31 01:15:35.816463 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.816441 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:35.868425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.868400 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:15:35.912371 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.912343 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:35.912558 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.912505 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngd57\" (UniqueName: \"kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:35.912558 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.912538 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:35.919589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.919564 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl"] Oct 31 01:15:35.923504 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.923486 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:35.923718 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.923701 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:15:35.972973 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:35.972946 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:15:36.013599 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.013571 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngd57\" (UniqueName: \"kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.013744 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.013607 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.013744 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.013668 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.014365 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.014338 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.016400 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.016377 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.074631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.074566 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngd57\" (UniqueName: \"kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57\") pod \"maas-api-revoke-sub-55a107ff-mv6vl\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.130493 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.130465 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:36.505489 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.505462 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl"] Oct 31 01:15:36.508304 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:15:36.508277 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61a5f21_d97a_4ee8_8ebb_27f5ca16dab2.slice/crio-3b6d1bab37966a1338e6fb076b7f395af0b68f0fd9cfaf13ea6851796ac8fb9e WatchSource:0}: Error finding container 3b6d1bab37966a1338e6fb076b7f395af0b68f0fd9cfaf13ea6851796ac8fb9e: Status 404 returned error can't find the container with id 3b6d1bab37966a1338e6fb076b7f395af0b68f0fd9cfaf13ea6851796ac8fb9e Oct 31 01:15:36.635822 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.635794 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:36.785335 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:36.785249 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:37.384776 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:37.384735 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" event={"ID":"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2","Type":"ContainerStarted","Data":"49027bb0dfa8ec77e5ccb9b02c94e8bba1c57fae922700c5118a69956e553bcb"} Oct 31 01:15:37.385167 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:37.384787 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" event={"ID":"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2","Type":"ContainerStarted","Data":"3b6d1bab37966a1338e6fb076b7f395af0b68f0fd9cfaf13ea6851796ac8fb9e"} Oct 31 01:15:37.499937 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:37.499885 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" podStartSLOduration=2.499869734 podStartE2EDuration="2.499869734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:37.499545221 +0000 UTC m=+4317.592835425" watchObservedRunningTime="2026-10-31 01:15:37.499869734 +0000 UTC m=+4317.593159964" Oct 31 01:15:38.391412 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:38.391375 5943 generic.go:356] "Generic (PLEG): container finished" podID="a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" containerID="49027bb0dfa8ec77e5ccb9b02c94e8bba1c57fae922700c5118a69956e553bcb" exitCode=0 Oct 31 01:15:38.391827 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:38.391421 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" event={"ID":"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2","Type":"ContainerDied","Data":"49027bb0dfa8ec77e5ccb9b02c94e8bba1c57fae922700c5118a69956e553bcb"} Oct 31 01:15:39.528482 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.528458 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:39.648050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.647984 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca\") pod \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " Oct 31 01:15:39.648050 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.648034 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57\" (UniqueName: \"kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57\") pod \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " Oct 31 01:15:39.648316 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.648092 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca\") pod \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\" (UID: \"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2\") " Oct 31 01:15:39.648525 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.648495 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca" pod "a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" (UID: "a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:39.650301 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.650268 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca" pod "a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" (UID: "a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:39.650449 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.650431 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57" pod "a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" (UID: "a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2"). InnerVolumeSpecName "kube-api-access-ngd57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.749137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.749096 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-xks-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.749137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.749129 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngd57\" (UniqueName: \"kubernetes.io/projected/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-kube-api-access-ngd57\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.749137 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:39.749139 5943 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2-maas-api-openshift-ca\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.410732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:40.410652 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" Oct 31 01:15:40.410732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:40.410669 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a107ff-mv6vl" event={"ID":"a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2","Type":"ContainerDied","Data":"3b6d1bab37966a1338e6fb076b7f395af0b68f0fd9cfaf13ea6851796ac8fb9e"} Oct 31 01:15:40.410938 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:40.410743 5943 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6d1bab37966a1338e6fb076b7f395af0b68f0fd9cfaf13ea6851796ac8fb9e" Oct 31 01:15:42.339225 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.339171 5943 scope.go:122] "RemoveContainer" containerID="2bc656a5660e7af6af1965f5f02c35c52544cd800a95ff64e0b280b696c8edfd" Oct 31 01:15:42.348556 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.348537 5943 scope.go:122] "RemoveContainer" containerID="8390ec50bbf9b3650334b34e43e87dbde8437ee5595018ffe23ac6042722d506" Oct 31 01:15:42.357327 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.357309 5943 scope.go:122] "RemoveContainer" containerID="aaa626ce1b8f2cf9bebaeded8f9b0e5252ce13c8a67e7fa146dca50daf31d152" Oct 31 01:15:42.365606 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.365590 5943 scope.go:122] "RemoveContainer" containerID="94378f3aa13ae1937eebebcdc47d5fbfb854fec4218a07945de8ff285b477aaa" Oct 31 01:15:42.373354 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.373327 5943 scope.go:122] "RemoveContainer" containerID="49f8d7a903bdbf8780592e068a2be55fb6953698f8205bfa1178b05aa070f889" Oct 31 01:15:42.381320 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.381296 5943 scope.go:122] "RemoveContainer" containerID="6957e625888ce551f8b07019d8c513843f7f167c923b66658f26ddc0319fe2db" Oct 31 01:15:42.389664 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:42.389646 5943 scope.go:122] "RemoveContainer" containerID="218bac3806717ea248ae67971bcb89d480534375e3da40cddcc856a3ccc04aed" Oct 31 01:15:44.237557 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.237505 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" containerID="cri-o://e7bdaa28af34e8f27219bef82faf04150d0c6ba61768a26ac7a74705c85b4614" gracePeriod=45 Oct 31 01:15:44.238398 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.238041 5943 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" containerID="cri-o://d1c464cc9c0de012812c939438b573b01b5689965046e52ce4dee6b546098f2b" gracePeriod=45 Oct 31 01:15:44.310500 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.310466 5943 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": dial tcp 10.134.0.46:8000: connect: connection refused" Oct 31 01:15:44.439074 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.439044 5943 generic.go:356] "Generic (PLEG): container finished" podID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerID="e7bdaa28af34e8f27219bef82faf04150d0c6ba61768a26ac7a74705c85b4614" exitCode=0 Oct 31 01:15:44.439250 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.439115 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" event={"ID":"f4c15de9-f253-4fa2-88f2-82c201288e2c","Type":"ContainerDied","Data":"e7bdaa28af34e8f27219bef82faf04150d0c6ba61768a26ac7a74705c85b4614"} Oct 31 01:15:44.441053 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.441023 5943 generic.go:356] "Generic (PLEG): container finished" podID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerID="d1c464cc9c0de012812c939438b573b01b5689965046e52ce4dee6b546098f2b" exitCode=0 Oct 31 01:15:44.441227 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.441097 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" event={"ID":"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc","Type":"ContainerDied","Data":"d1c464cc9c0de012812c939438b573b01b5689965046e52ce4dee6b546098f2b"} Oct 31 01:15:44.499724 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.499698 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 01:15:44.502755 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.502735 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 01:15:44.597136 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597104 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg\" (UniqueName: \"kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg\") pod \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " Oct 31 01:15:44.597334 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597140 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir\") pod \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " Oct 31 01:15:44.597334 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597234 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir\") pod \"f4c15de9-f253-4fa2-88f2-82c201288e2c\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " Oct 31 01:15:44.597334 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597268 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home\") pod \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " Oct 31 01:15:44.597480 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597411 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache\") pod \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " Oct 31 01:15:44.597480 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597452 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm\") pod \"f4c15de9-f253-4fa2-88f2-82c201288e2c\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " Oct 31 01:15:44.597589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597517 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm\") pod \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " Oct 31 01:15:44.597589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597515 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir" pod "f4c15de9-f253-4fa2-88f2-82c201288e2c" (UID: "f4c15de9-f253-4fa2-88f2-82c201288e2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.597589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597541 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home" pod "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" (UID: "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.597589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597530 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir" pod "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" (UID: "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.597589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597556 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs\") pod \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\" (UID: \"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc\") " Oct 31 01:15:44.597589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597582 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home\") pod \"f4c15de9-f253-4fa2-88f2-82c201288e2c\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " Oct 31 01:15:44.597935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597634 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c\" (UniqueName: \"kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c\") pod \"f4c15de9-f253-4fa2-88f2-82c201288e2c\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " Oct 31 01:15:44.597935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597694 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache\") pod \"f4c15de9-f253-4fa2-88f2-82c201288e2c\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " Oct 31 01:15:44.597935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597721 5943 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs\") pod \"f4c15de9-f253-4fa2-88f2-82c201288e2c\" (UID: \"f4c15de9-f253-4fa2-88f2-82c201288e2c\") " Oct 31 01:15:44.597935 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.597849 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home" pod "f4c15de9-f253-4fa2-88f2-82c201288e2c" (UID: "f4c15de9-f253-4fa2-88f2-82c201288e2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.598131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.598064 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.598131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.598085 5943 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-tmp-dir\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.598131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.598099 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.598131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.598112 5943 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-home\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.598397 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.598376 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache" pod "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" (UID: "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.598462 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.598445 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache" pod "f4c15de9-f253-4fa2-88f2-82c201288e2c" (UID: "f4c15de9-f253-4fa2-88f2-82c201288e2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.600526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.600505 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg" pod "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" (UID: "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc"). InnerVolumeSpecName "kube-api-access-zcsjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:44.600636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.600524 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm" pod "f4c15de9-f253-4fa2-88f2-82c201288e2c" (UID: "f4c15de9-f253-4fa2-88f2-82c201288e2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.600636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.600527 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs" pod "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" (UID: "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:44.600636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.600541 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm" pod "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" (UID: "88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.600636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.600608 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c" pod "f4c15de9-f253-4fa2-88f2-82c201288e2c" (UID: "f4c15de9-f253-4fa2-88f2-82c201288e2c"). InnerVolumeSpecName "kube-api-access-5gb5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:44.600636 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.600619 5943 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs" pod "f4c15de9-f253-4fa2-88f2-82c201288e2c" (UID: "f4c15de9-f253-4fa2-88f2-82c201288e2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:44.698763 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698725 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gb5c\" (UniqueName: \"kubernetes.io/projected/f4c15de9-f253-4fa2-88f2-82c201288e2c-kube-api-access-5gb5c\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.698763 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698764 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.699034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698782 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c15de9-f253-4fa2-88f2-82c201288e2c-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.699034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698797 5943 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsjg\" (UniqueName: \"kubernetes.io/projected/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-kube-api-access-zcsjg\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.699034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698810 5943 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-model-cache\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.699034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698822 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c15de9-f253-4fa2-88f2-82c201288e2c-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.699034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698835 5943 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-dshm\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.699034 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:44.698847 5943 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc-tls-certs\") on node \"ip-10-0-143-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:45.448238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.448209 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" Oct 31 01:15:45.448725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.448239 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf" event={"ID":"f4c15de9-f253-4fa2-88f2-82c201288e2c","Type":"ContainerDied","Data":"c45f43ca80c43ef702e09e4ae10f86ca9350fa398da3e1a36a9cda0fa4307003"} Oct 31 01:15:45.448725 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.448296 5943 scope.go:122] "RemoveContainer" containerID="e7bdaa28af34e8f27219bef82faf04150d0c6ba61768a26ac7a74705c85b4614" Oct 31 01:15:45.449791 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.449768 5943 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" Oct 31 01:15:45.449791 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.449780 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg" event={"ID":"88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc","Type":"ContainerDied","Data":"98ead338aefdefde75bff6d48375c112dd6765d3a00caaef1191cfa589aa7763"} Oct 31 01:15:45.458828 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.458810 5943 scope.go:122] "RemoveContainer" containerID="d1c464cc9c0de012812c939438b573b01b5689965046e52ce4dee6b546098f2b" Oct 31 01:15:45.611560 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.611518 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf"] Oct 31 01:15:45.615160 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.615134 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/premium-w4-0144bd-kserve-bcb9d9c85-2d7rf"] Oct 31 01:15:45.726366 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.726335 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg"] Oct 31 01:15:45.731342 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:45.731310 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/simulator-w4-0144bd-kserve-59bd5b7d68-nkplg"] Oct 31 01:15:46.040318 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:46.040282 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" path="/var/lib/kubelet/pods/88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc/volumes" Oct 31 01:15:46.040692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:46.040679 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" path="/var/lib/kubelet/pods/f4c15de9-f253-4fa2-88f2-82c201288e2c/volumes" Oct 31 01:15:50.572245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:50.572132 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:52.075403 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:52.075369 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:52.346876 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:52.346791 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:15:52.350080 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:52.350037 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:53.029798 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:53.029733 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:57.474486 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:15:57.474447 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:15:59.610473 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:59.610436 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:59.756768 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:59.756733 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:59.989880 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:15:59.989776 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:03.258029 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:03.257990 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:04.323879 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:04.323847 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d2fw9_7dc29b08-40a6-4473-90e8-48ac49e661d3/limitador/0.log" Oct 31 01:16:04.595277 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:04.595198 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hlvcx_4640a92d-1063-4a9f-afbb-6221ba3bc084/manager/0.log" Oct 31 01:16:05.716345 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:05.716307 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:05.868114 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:05.868072 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:06.057631 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:06.057598 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d2fw9"] Oct 31 01:16:06.110238 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:06.110210 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:16:06.492534 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:06.492443 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:16:06.935984 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:06.935947 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:07.032649 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:07.032610 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:16:07.520905 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:07.520865 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:16:08.333310 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:08.333267 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:16:12.725161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:12.725122 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:20.386467 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:20.386434 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:16:20.772266 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:20.772226 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:16:21.364161 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:21.364121 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:16:21.850413 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:21.850385 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:16:22.679589 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:22.679554 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:16:23.708607 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:23.708574 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-009adcdb-89pw8_b397ab0c-73f9-4ef2-a403-14d3ed5fcb05/revoke-keys/0.log" Oct 31 01:16:23.933625 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:23.933587 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-06a462b8-6tbrl_8fc2003a-d129-4d24-9085-503fd6974526/revoke-keys/0.log" Oct 31 01:16:24.166930 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:24.166900 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-46d4bcfd-9lq95_1aba2032-5797-4934-ab3b-42c7095d6c0a/revoke-keys/0.log" Oct 31 01:16:24.400742 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:24.400708 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-55a107ff-mv6vl_a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2/revoke-keys/0.log" Oct 31 01:16:24.625363 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:24.625336 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e024b754-6w9dp_15e2439b-3459-49c6-9f15-46e960ca98b1/revoke-keys/0.log" Oct 31 01:16:27.474993 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:16:27.474952 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:16:28.931437 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:28.931408 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d2fw9_7dc29b08-40a6-4473-90e8-48ac49e661d3/limitador/0.log" Oct 31 01:16:29.168307 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:29.168273 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hlvcx_4640a92d-1063-4a9f-afbb-6221ba3bc084/manager/0.log" Oct 31 01:16:30.926965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:30.926937 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-j7qlx_8a21f85d-c34b-4149-a0c1-2ba949b1b120/kube-auth-proxy/0.log" Oct 31 01:16:31.428937 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:31.428906 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk_36de5579-6d9a-4a29-b77f-2d417d9f7dc5/istio-proxy/0.log" Oct 31 01:16:36.590118 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:36.590077 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:42.446119 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:42.446088 5943 scope.go:122] "RemoveContainer" containerID="0ffa06349455123b32a040f1986135bcc1bd727a6073915c7c6418c7df766bbd" Oct 31 01:16:42.455425 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:42.455405 5943 scope.go:122] "RemoveContainer" containerID="0d708b6846d3a97ecedff9a25a5b45355259e34ea599eb686651f54468288628" Oct 31 01:16:42.463699 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:42.463680 5943 scope.go:122] "RemoveContainer" containerID="a0c0f32d0ea50830c23d62120d428dbd72a5da3103790dc6452ad7fce0eb24da" Oct 31 01:16:45.449448 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.449406 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:45.532538 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532500 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b8w7c/must-gather-zqn98"] Oct 31 01:16:45.532910 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532898 5943 cpu_manager.go:414] "containerMap: removing container" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" Oct 31 01:16:45.532958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532911 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" Oct 31 01:16:45.532958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532931 5943 cpu_manager.go:414] "containerMap: removing container" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" Oct 31 01:16:45.532958 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532937 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" Oct 31 01:16:45.533060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532962 5943 cpu_manager.go:414] "containerMap: removing container" podUID="a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" containerName="revoke-keys" Oct 31 01:16:45.533060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532966 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" containerName="revoke-keys" Oct 31 01:16:45.533060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532977 5943 cpu_manager.go:414] "containerMap: removing container" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:16:45.533060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.532981 5943 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4ad16f-1be6-4cb1-a9a6-e02ce871e6fc" containerName="cleanup" Oct 31 01:16:45.533060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.533047 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61a5f21-d97a-4ee8-8ebb-27f5ca16dab2" containerName="revoke-keys" Oct 31 01:16:45.533060 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.533059 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="88533ac8-d33e-4d1a-a3e2-b47d0da5f2bc" containerName="main" Oct 31 01:16:45.533278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.533065 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b91f93-ae8e-418c-ac6f-53a444084621" containerName="istio-proxy" Oct 31 01:16:45.533278 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.533071 5943 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c15de9-f253-4fa2-88f2-82c201288e2c" containerName="main" Oct 31 01:16:45.537917 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.537899 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.590413 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.590381 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0f10edd-ad67-419c-8988-d4d1f86c98f6-must-gather-output\") pod \"must-gather-zqn98\" (UID: \"a0f10edd-ad67-419c-8988-d4d1f86c98f6\") " pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.590558 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.590495 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hx8\" (UniqueName: \"kubernetes.io/projected/a0f10edd-ad67-419c-8988-d4d1f86c98f6-kube-api-access-f8hx8\") pod \"must-gather-zqn98\" (UID: \"a0f10edd-ad67-419c-8988-d4d1f86c98f6\") " pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.643726 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.643686 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b8w7c\"/\"openshift-service-ca.crt\"" Oct 31 01:16:45.643726 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.643693 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b8w7c\"/\"kube-root-ca.crt\"" Oct 31 01:16:45.644590 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.644570 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b8w7c\"/\"default-dockercfg-65tqq\"" Oct 31 01:16:45.646062 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.646042 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b8w7c/must-gather-zqn98"] Oct 31 01:16:45.667969 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.667927 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:45.691965 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.691932 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0f10edd-ad67-419c-8988-d4d1f86c98f6-must-gather-output\") pod \"must-gather-zqn98\" (UID: \"a0f10edd-ad67-419c-8988-d4d1f86c98f6\") " pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.692153 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.692000 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hx8\" (UniqueName: \"kubernetes.io/projected/a0f10edd-ad67-419c-8988-d4d1f86c98f6-kube-api-access-f8hx8\") pod \"must-gather-zqn98\" (UID: \"a0f10edd-ad67-419c-8988-d4d1f86c98f6\") " pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.692399 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.692378 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0f10edd-ad67-419c-8988-d4d1f86c98f6-must-gather-output\") pod \"must-gather-zqn98\" (UID: \"a0f10edd-ad67-419c-8988-d4d1f86c98f6\") " pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.754446 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.754373 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hx8\" (UniqueName: \"kubernetes.io/projected/a0f10edd-ad67-419c-8988-d4d1f86c98f6-kube-api-access-f8hx8\") pod \"must-gather-zqn98\" (UID: \"a0f10edd-ad67-419c-8988-d4d1f86c98f6\") " pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:45.852297 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:45.852266 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b8w7c/must-gather-zqn98" Oct 31 01:16:46.223388 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:46.223361 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b8w7c/must-gather-zqn98"] Oct 31 01:16:46.226153 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:16:46.226121 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0f10edd_ad67_419c_8988_d4d1f86c98f6.slice/crio-73236f2e8cfb4be8020c181b8e7bb1dd3a389eaa1815f0dfbc694b3b57b25422 WatchSource:0}: Error finding container 73236f2e8cfb4be8020c181b8e7bb1dd3a389eaa1815f0dfbc694b3b57b25422: Status 404 returned error can't find the container with id 73236f2e8cfb4be8020c181b8e7bb1dd3a389eaa1815f0dfbc694b3b57b25422 Oct 31 01:16:46.228121 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:46.228104 5943 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:16:46.781383 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:46.781341 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8w7c/must-gather-zqn98" event={"ID":"a0f10edd-ad67-419c-8988-d4d1f86c98f6","Type":"ContainerStarted","Data":"73236f2e8cfb4be8020c181b8e7bb1dd3a389eaa1815f0dfbc694b3b57b25422"} Oct 31 01:16:52.826486 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:52.826447 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8w7c/must-gather-zqn98" event={"ID":"a0f10edd-ad67-419c-8988-d4d1f86c98f6","Type":"ContainerStarted","Data":"e7235e8ca80a3cdd532e7a059f2ca63fb6cfafd5b1b108dc36378a305adfe9a7"} Oct 31 01:16:52.826895 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:52.826505 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b8w7c/must-gather-zqn98" event={"ID":"a0f10edd-ad67-419c-8988-d4d1f86c98f6","Type":"ContainerStarted","Data":"25a86e9a7f3a44fc353459f4611d603e5abdf05742643a8f3c30db6589abc7ed"} Oct 31 01:16:52.987465 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:52.987409 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b8w7c/must-gather-zqn98" podStartSLOduration=1.835523159 podStartE2EDuration="7.987393803s" totalImagesPullingTime="6.151870644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:52.939870229 +0000 UTC m=+4393.033160415" watchObservedRunningTime="2026-10-31 01:16:52.987393803 +0000 UTC m=+4393.080683990" Oct 31 01:16:55.465905 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:55.465869 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-68spz_1bc7e49e-8890-47c1-a64f-25328161fde3/global-pull-secret-syncer/0.log" Oct 31 01:16:57.004859 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:57.004820 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:16:57.485554 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:16:57.485458 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:16:57.782597 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:57.782559 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:16:57.897705 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:57.897669 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:16:58.146696 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:58.146601 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" Oct 31 01:16:58.277908 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:58.277879 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:16:58.974271 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:58.974212 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:16:59.958205 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:59.958118 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:16:59.958765 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:16:59.958656 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:17:00.206856 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:00.206817 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:17:00.270000 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:00.269965 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-53.us-west-1.compute.internal_8844070afaf498134d5a363201a59ac4/haproxy/0.log" Oct 31 01:17:00.541364 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:00.541260 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:17:02.327724 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:02.327690 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:17:03.631369 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:03.631335 5943 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hztx_386772ab-964a-485d-9489-75e74e1fe1de/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:03.725188 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:03.725141 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:17:04.462970 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:04.462930 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:17:04.593510 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:04.593480 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:17:04.938773 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:04.938738 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:17:05.587794 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.587753 5943 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb"] Oct 31 01:17:05.594892 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.594856 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.647072 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.647033 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m9n5w\"/\"kube-root-ca.crt\"" Oct 31 01:17:05.647282 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.647077 5943 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m9n5w\"/\"openshift-service-ca.crt\"" Oct 31 01:17:05.647282 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.647131 5943 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m9n5w\"/\"default-dockercfg-2hfcm\"" Oct 31 01:17:05.699428 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.699395 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb"] Oct 31 01:17:05.776956 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.776918 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-podres\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.777353 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.777290 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2lq\" (UniqueName: \"kubernetes.io/projected/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-kube-api-access-ns2lq\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.777353 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.777364 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-lib-modules\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.777594 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.777435 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-sys\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.777594 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.777533 5943 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-proc\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882096 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882001 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2lq\" (UniqueName: \"kubernetes.io/projected/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-kube-api-access-ns2lq\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882304 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882282 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-lib-modules\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882568 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882541 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-sys\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882799 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882767 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-sys\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882799 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882787 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-proc\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882853 5943 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-podres\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.882942 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882696 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-lib-modules\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.883054 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.882853 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-proc\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.883115 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.883020 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-podres\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:05.945264 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:05.945233 5943 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2lq\" (UniqueName: \"kubernetes.io/projected/5d02c3cb-7d83-42d7-af60-6d7fb9967fd1-kube-api-access-ns2lq\") pod \"perf-node-gather-daemonset-z6xwb\" (UID: \"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:06.213675 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.213581 5943 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:06.387209 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.387162 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:17:06.611433 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.611397 5943 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb"] Oct 31 01:17:06.614780 ip-10-0-143-53 kubenswrapper[5943]: W1031 01:17:06.614757 5943 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d02c3cb_7d83_42d7_af60_6d7fb9967fd1.slice/crio-ea93676b59f494b48727caa6ad761d98592e472fa2db0d5df6410d5590e7ac11 WatchSource:0}: Error finding container ea93676b59f494b48727caa6ad761d98592e472fa2db0d5df6410d5590e7ac11: Status 404 returned error can't find the container with id ea93676b59f494b48727caa6ad761d98592e472fa2db0d5df6410d5590e7ac11 Oct 31 01:17:06.796599 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.796569 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:17:06.916630 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.916546 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" event={"ID":"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1","Type":"ContainerStarted","Data":"a011617e88d7247924982eadaddfe902b30ccd5675f8b33a4e99c7de32d70c24"} Oct 31 01:17:06.916630 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.916599 5943 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" event={"ID":"5d02c3cb-7d83-42d7-af60-6d7fb9967fd1","Type":"ContainerStarted","Data":"ea93676b59f494b48727caa6ad761d98592e472fa2db0d5df6410d5590e7ac11"} Oct 31 01:17:06.916826 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:06.916757 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:07.034556 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:07.034470 5943 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" podStartSLOduration=2.034453215 podStartE2EDuration="2.034453215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:07.033927889 +0000 UTC m=+4407.127218098" watchObservedRunningTime="2026-10-31 01:17:07.034453215 +0000 UTC m=+4407.127743404" Oct 31 01:17:07.395795 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:07.395764 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d2fw9_7dc29b08-40a6-4473-90e8-48ac49e661d3/limitador/0.log" Oct 31 01:17:07.474820 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:07.474790 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:17:07.599878 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:07.599845 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:17:07.952099 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:07.952069 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hlvcx_4640a92d-1063-4a9f-afbb-6221ba3bc084/manager/0.log" Oct 31 01:17:07.973551 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:07.973510 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:17:09.057131 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:09.057076 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp"] Oct 31 01:17:09.105731 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:09.105701 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e024b754-6w9dp"] Oct 31 01:17:09.308806 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:09.308734 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:17:10.040479 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.040434 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e2439b-3459-49c6-9f15-46e960ca98b1" path="/var/lib/kubelet/pods/15e2439b-3459-49c6-9f15-46e960ca98b1/volumes" Oct 31 01:17:10.046149 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.046121 5943 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl"] Oct 31 01:17:10.068849 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.068814 5943 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a462b8-6tbrl"] Oct 31 01:17:10.122526 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.122492 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:17:10.531671 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.531635 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/alertmanager/0.log" Oct 31 01:17:10.839097 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.839002 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:17:10.951967 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.951937 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/config-reloader/0.log" Oct 31 01:17:10.969692 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:10.969657 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:17:11.320193 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:11.320142 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:17:11.371405 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:11.371370 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/kube-rbac-proxy-web/0.log" Oct 31 01:17:11.790780 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:11.790746 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/kube-rbac-proxy/0.log" Oct 31 01:17:12.043962 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:12.043867 5943 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc2003a-d129-4d24-9085-503fd6974526" path="/var/lib/kubelet/pods/8fc2003a-d129-4d24-9085-503fd6974526/volumes" Oct 31 01:17:12.211334 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:12.211301 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/kube-rbac-proxy-metric/0.log" Oct 31 01:17:12.638245 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:12.638212 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/prom-label-proxy/0.log" Oct 31 01:17:12.744235 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:12.744201 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:17:12.901761 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:12.901675 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nmcf6_c2884d99-17e0-486e-ba7e-fec1605f05f2/manager/0.log" Oct 31 01:17:12.942671 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:12.942643 5943 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-z6xwb" Oct 31 01:17:13.061005 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:13.060974 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63e93e59-7eef-46cc-8acc-9d5bb3871736/init-config-reloader/0.log" Oct 31 01:17:13.583863 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:13.583823 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8ldm6_b5e2db63-a291-4143-8832-9d64a03e8484/manager/0.log" Oct 31 01:17:13.711295 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:13.711265 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/0.log" Oct 31 01:17:13.931658 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:13.931582 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4txlr_a61bcee6-5eb5-44b0-af5b-81bf788bf136/kube-state-metrics/0.log" Oct 31 01:17:14.041972 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:14.041944 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f54bdd8cd-ncrz5_72141891-b5bf-4a1f-9fbe-aebe6e07bcc0/manager/1.log" Oct 31 01:17:14.338246 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:14.338219 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4txlr_a61bcee6-5eb5-44b0-af5b-81bf788bf136/kube-rbac-proxy-main/0.log" Oct 31 01:17:14.757009 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:14.756979 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4txlr_a61bcee6-5eb5-44b0-af5b-81bf788bf136/kube-rbac-proxy-self/0.log" Oct 31 01:17:15.182880 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:15.182799 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86d6c7656b-2hqmr_822e1679-db28-423e-91c0-0dd4cf714902/metrics-server/0.log" Oct 31 01:17:15.413744 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:15.413712 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5s5vk_de47dd60-f7d9-4034-93a5-128c5fa025a7/manager/0.log" Oct 31 01:17:15.605939 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:15.605909 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-st7jb_888e451d-b53f-487c-bd01-c90a6ad9bf8a/monitoring-plugin/0.log" Oct 31 01:17:16.044373 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:16.044336 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67jm8_a0a53051-3a82-499c-bd29-9ccf93a535bc/node-exporter/0.log" Oct 31 01:17:16.467217 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:16.467101 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67jm8_a0a53051-3a82-499c-bd29-9ccf93a535bc/kube-rbac-proxy/0.log" Oct 31 01:17:16.888255 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:16.888224 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67jm8_a0a53051-3a82-499c-bd29-9ccf93a535bc/init-textfile/0.log" Oct 31 01:17:17.593732 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:17.593698 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-j7qlx_8a21f85d-c34b-4149-a0c1-2ba949b1b120/kube-auth-proxy/0.log" Oct 31 01:17:18.032888 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:18.032857 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk_36de5579-6d9a-4a29-b77f-2d417d9f7dc5/istio-proxy/0.log" Oct 31 01:17:20.032529 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:20.032450 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s8ddg_1ce2530e-5ad4-4eec-ad2c-58bca81cae53/kube-rbac-proxy-main/0.log" Oct 31 01:17:20.202266 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:20.202235 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-j7qlx_8a21f85d-c34b-4149-a0c1-2ba949b1b120/kube-auth-proxy/0.log" Oct 31 01:17:20.453148 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:20.453041 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s8ddg_1ce2530e-5ad4-4eec-ad2c-58bca81cae53/kube-rbac-proxy-self/0.log" Oct 31 01:17:20.648628 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:20.648590 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-9wkgk_36de5579-6d9a-4a29-b77f-2d417d9f7dc5/istio-proxy/0.log" Oct 31 01:17:20.884708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:20.884671 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s8ddg_1ce2530e-5ad4-4eec-ad2c-58bca81cae53/openshift-state-metrics/0.log" Oct 31 01:17:21.318505 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:21.318461 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/prometheus/0.log" Oct 31 01:17:21.738588 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:21.738509 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/config-reloader/0.log" Oct 31 01:17:22.158582 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:22.158554 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/thanos-sidecar/0.log" Oct 31 01:17:22.578145 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:22.578111 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/kube-rbac-proxy-web/0.log" Oct 31 01:17:23.002645 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:23.002566 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/kube-rbac-proxy/0.log" Oct 31 01:17:23.421871 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:23.421825 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/kube-rbac-proxy-thanos/0.log" Oct 31 01:17:23.840916 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:23.840870 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_487a23e3-9048-48fb-be8f-f29fcbaf8de4/init-config-reloader/0.log" Oct 31 01:17:25.582709 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:25.582682 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c8d84c7d8-5v8zs_504be407-0ef5-4f9f-b6f5-5674004659b8/telemeter-client/0.log" Oct 31 01:17:25.605225 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:25.605195 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d2fw9_7dc29b08-40a6-4473-90e8-48ac49e661d3/limitador/0.log" Oct 31 01:17:25.887835 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:25.887753 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hlvcx_4640a92d-1063-4a9f-afbb-6221ba3bc084/manager/0.log" Oct 31 01:17:26.006424 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:26.005533 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c8d84c7d8-5v8zs_504be407-0ef5-4f9f-b6f5-5674004659b8/reload/0.log" Oct 31 01:17:26.425152 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:26.425113 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c8d84c7d8-5v8zs_504be407-0ef5-4f9f-b6f5-5674004659b8/kube-rbac-proxy/0.log" Oct 31 01:17:26.853695 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:26.853666 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b47fd846-94zxw_2163faff-91f0-450a-aa4e-66336810549e/thanos-query/0.log" Oct 31 01:17:27.272708 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:27.272668 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b47fd846-94zxw_2163faff-91f0-450a-aa4e-66336810549e/kube-rbac-proxy-web/0.log" Oct 31 01:17:27.475482 ip-10-0-143-53 kubenswrapper[5943]: E1031 01:17:27.475444 5943 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q4dds" containerName="extractor" Oct 31 01:17:27.694893 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:27.694818 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b47fd846-94zxw_2163faff-91f0-450a-aa4e-66336810549e/kube-rbac-proxy/0.log" Oct 31 01:17:28.147609 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:28.147577 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b47fd846-94zxw_2163faff-91f0-450a-aa4e-66336810549e/prom-label-proxy/0.log" Oct 31 01:17:28.526602 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:28.526575 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d2fw9_7dc29b08-40a6-4473-90e8-48ac49e661d3/limitador/0.log" Oct 31 01:17:28.566071 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:28.566040 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b47fd846-94zxw_2163faff-91f0-450a-aa4e-66336810549e/kube-rbac-proxy-rules/0.log" Oct 31 01:17:28.813876 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:28.813775 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hlvcx_4640a92d-1063-4a9f-afbb-6221ba3bc084/manager/0.log" Oct 31 01:17:28.984648 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:28.984615 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b47fd846-94zxw_2163faff-91f0-450a-aa4e-66336810549e/kube-rbac-proxy-metrics/0.log" Oct 31 01:17:30.625579 ip-10-0-143-53 kubenswrapper[5943]: I1031 01:17:30.625541 5943 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-j7qlx_8a21f85d-c34b-4149-a0c1-2ba949b1b120/kube-auth-proxy/0.log"